2024-03-25T16:09:01.877 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T16:09:01.878 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_amathuri@teuthology --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621249 --verbose --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621249/orig.config.yaml 2024-03-25T16:09:01.920 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T16:09:02.008 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621249 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621249' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi066.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHiFRiyARlFOX6EzyMAmT7g0HngrubTPU4jsSfsyWrlEflazx4IQRo9ara2llBy4NEOGryTG/sLr+mPufh99/n8= smithi193.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDCcALhUKDoQ73A9zVvrTXTBClHg7qAmouPpjds8k2OzS4rF5FKG4D5lDjy5mM8V+0RU7Lck9wnVivw9YWcRFI8= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T16:09:02.009 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T16:09:02.009 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T16:09:02.009 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T16:09:02.011 INFO:teuthology.task.internal:Checking packages... 2024-03-25T16:09:02.035 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T16:09:02.035 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T16:09:02.035 INFO:teuthology.packaging:ref: None 2024-03-25T16:09:02.035 INFO:teuthology.packaging:tag: None 2024-03-25T16:09:02.035 INFO:teuthology.packaging:branch: main 2024-03-25T16:09:02.035 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:09:02.035 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T16:09:02.266 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T16:09:02.311 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T16:09:02.318 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T16:09:02.318 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T16:09:02.370 INFO:teuthology.task.internal:Saving configuration 2024-03-25T16:09:02.386 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T16:09:02.479 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T16:09:02.506 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi066.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621249', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:02:22.837681', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHiFRiyARlFOX6EzyMAmT7g0HngrubTPU4jsSfsyWrlEflazx4IQRo9ara2llBy4NEOGryTG/sLr+mPufh99/n8='} 2024-03-25T16:09:02.527 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621249', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:02:22.835849', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDCcALhUKDoQ73A9zVvrTXTBClHg7qAmouPpjds8k2OzS4rF5FKG4D5lDjy5mM8V+0RU7Lck9wnVivw9YWcRFI8='} 2024-03-25T16:09:02.527 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T16:09:02.534 INFO:teuthology.task.internal:roles: ubuntu@smithi066.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T16:09:02.534 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T16:09:02.534 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T16:09:02.603 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd7df4980d0>, signals=[15]) 2024-03-25T16:09:02.604 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T16:09:02.611 INFO:teuthology.task.internal:Opening connections... 2024-03-25T16:09:02.611 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi066.front.sepia.ceph.com 2024-03-25T16:09:02.614 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:09:02.765 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi193.front.sepia.ceph.com 2024-03-25T16:09:02.769 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:09:02.857 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T16:09:02.864 DEBUG:teuthology.orchestra.run.smithi066:> uname -m 2024-03-25T16:09:02.884 INFO:teuthology.orchestra.run.smithi066.stdout:x86_64 2024-03-25T16:09:02.884 DEBUG:teuthology.orchestra.run.smithi066:> cat /etc/os-release 2024-03-25T16:09:02.942 INFO:teuthology.orchestra.run.smithi066.stdout:NAME="CentOS Stream" 2024-03-25T16:09:02.942 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION="9" 2024-03-25T16:09:02.942 INFO:teuthology.orchestra.run.smithi066.stdout:ID="centos" 2024-03-25T16:09:02.942 INFO:teuthology.orchestra.run.smithi066.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:09:02.942 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION_ID="9" 2024-03-25T16:09:02.942 INFO:teuthology.orchestra.run.smithi066.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:09:02.942 INFO:teuthology.orchestra.run.smithi066.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:09:02.942 INFO:teuthology.orchestra.run.smithi066.stdout:ANSI_COLOR="0;31" 2024-03-25T16:09:02.942 INFO:teuthology.orchestra.run.smithi066.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:09:02.943 INFO:teuthology.orchestra.run.smithi066.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:09:02.943 INFO:teuthology.orchestra.run.smithi066.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:09:02.943 INFO:teuthology.orchestra.run.smithi066.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:09:02.943 INFO:teuthology.orchestra.run.smithi066.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:09:02.943 INFO:teuthology.orchestra.run.smithi066.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:09:02.943 INFO:teuthology.lock.ops:Updating smithi066.front.sepia.ceph.com on lock server 2024-03-25T16:09:02.973 DEBUG:teuthology.orchestra.run.smithi193:> uname -m 2024-03-25T16:09:02.990 INFO:teuthology.orchestra.run.smithi193.stdout:x86_64 2024-03-25T16:09:02.991 DEBUG:teuthology.orchestra.run.smithi193:> cat /etc/os-release 2024-03-25T16:09:03.046 INFO:teuthology.orchestra.run.smithi193.stdout:NAME="CentOS Stream" 2024-03-25T16:09:03.046 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION="9" 2024-03-25T16:09:03.046 INFO:teuthology.orchestra.run.smithi193.stdout:ID="centos" 2024-03-25T16:09:03.046 INFO:teuthology.orchestra.run.smithi193.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:09:03.047 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION_ID="9" 2024-03-25T16:09:03.047 INFO:teuthology.orchestra.run.smithi193.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:09:03.047 INFO:teuthology.orchestra.run.smithi193.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:09:03.047 INFO:teuthology.orchestra.run.smithi193.stdout:ANSI_COLOR="0;31" 2024-03-25T16:09:03.047 INFO:teuthology.orchestra.run.smithi193.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:09:03.047 INFO:teuthology.orchestra.run.smithi193.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:09:03.047 INFO:teuthology.orchestra.run.smithi193.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:09:03.047 INFO:teuthology.orchestra.run.smithi193.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:09:03.047 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:09:03.047 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:09:03.047 INFO:teuthology.lock.ops:Updating smithi193.front.sepia.ceph.com on lock server 2024-03-25T16:09:03.064 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T16:09:03.074 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T16:09:03.079 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T16:09:03.080 DEBUG:teuthology.orchestra.run.smithi066:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:09:03.083 DEBUG:teuthology.orchestra.run.smithi193:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:09:03.102 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T16:09:03.109 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T16:09:03.109 DEBUG:teuthology.orchestra.run.smithi066:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:09:03.140 DEBUG:teuthology.orchestra.run.smithi193:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:09:03.233 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T16:09:03.288 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T16:09:03.306 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T16:09:03.307 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T16:09:03.307 DEBUG:teuthology.orchestra.run.smithi066:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:09:03.308 DEBUG:teuthology.orchestra.run.smithi193:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:09:03.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:09:03.323 DEBUG:teuthology.orchestra.run.smithi193:> uname -r 2024-03-25T16:09:03.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:09:03.324 DEBUG:teuthology.orchestra.run.smithi066:> uname -r 2024-03-25T16:09:03.380 INFO:teuthology.orchestra.run.smithi193.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:09:03.380 INFO:teuthology.task.kernel:Running kernel on smithi193: 5.14.0-361.el9.x86_64 2024-03-25T16:09:03.380 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum install -y kernel 2024-03-25T16:09:03.383 INFO:teuthology.orchestra.run.smithi066.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:09:03.383 INFO:teuthology.task.kernel:Running kernel on smithi066: 5.14.0-361.el9.x86_64 2024-03-25T16:09:03.384 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum install -y kernel 2024-03-25T16:09:05.629 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-25T16:09:05.805 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-25T16:09:07.238 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-03-25T16:09:07.284 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS-9 - BaseOS 8.0 MB/s | 2.0 MB 00:00 2024-03-25T16:09:07.987 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - BaseOS 22 MB/s | 8.0 MB 00:00 2024-03-25T16:09:08.580 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - BaseOS 7.9 MB/s | 8.0 MB 00:01 2024-03-25T16:09:09.730 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - AppStream 31 MB/s | 19 MB 00:00 2024-03-25T16:09:11.046 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-03-25T16:09:14.295 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - CRB 7.1 MB/s | 6.1 MB 00:00 2024-03-25T16:09:15.149 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - CRB 9.1 MB/s | 6.1 MB 00:00 2024-03-25T16:09:15.798 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 16 kB 00:00 2024-03-25T16:09:16.547 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 16 kB 00:00 2024-03-25T16:09:17.035 INFO:teuthology.orchestra.run.smithi066.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-03-25T16:09:22.066 INFO:teuthology.orchestra.run.smithi066.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-25T16:09:24.282 INFO:teuthology.orchestra.run.smithi066.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:09:24.282 INFO:teuthology.orchestra.run.smithi066.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:09:24.378 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:09:24.390 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:09:24.390 INFO:teuthology.orchestra.run.smithi066.stdout: Package Architecture Version Repository Size 2024-03-25T16:09:24.390 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:09:24.390 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T16:09:24.391 INFO:teuthology.orchestra.run.smithi066.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:09:24.391 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-25T16:09:24.391 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:09:24.391 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:09:24.391 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:09:24.391 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:09:24.391 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T16:09:24.391 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:09:24.391 INFO:teuthology.orchestra.run.smithi066.stdout:Install 4 Packages 2024-03-25T16:09:24.391 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:09:24.391 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 76 M 2024-03-25T16:09:24.391 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 123 M 2024-03-25T16:09:24.391 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T16:09:24.653 INFO:teuthology.orchestra.run.smithi066.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 314 kB/s | 44 kB 00:00 2024-03-25T16:09:26.976 INFO:teuthology.orchestra.run.smithi066.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 6.0 MB/s | 15 MB 00:02 2024-03-25T16:09:27.718 INFO:teuthology.orchestra.run.smithi066.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 9.0 MB/s | 27 MB 00:03 2024-03-25T16:09:28.335 INFO:teuthology.orchestra.run.smithi066.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 8.7 MB/s | 33 MB 00:03 2024-03-25T16:09:28.335 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:09:28.335 INFO:teuthology.orchestra.run.smithi066.stdout:Total 19 MB/s | 76 MB 00:03 2024-03-25T16:09:29.024 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T16:09:29.439 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T16:09:29.440 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T16:09:30.471 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T16:09:30.558 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T16:09:33.069 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T16:09:33.696 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:09:33.793 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:09:36.657 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:09:36.794 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:09:41.874 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:09:41.920 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:09:47.014 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:10:17.631 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:10:17.631 INFO:teuthology.orchestra.run.smithi066.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:10:17.631 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:10:17.677 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:10:18.084 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:10:18.084 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:10:18.084 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:10:18.084 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:10:18.515 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:10:18.515 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:10:18.516 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T16:10:18.516 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:10:18.516 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:10:18.516 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:10:18.516 INFO:teuthology.orchestra.run.smithi066.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:10:18.516 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:10:18.516 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T16:10:18.818 DEBUG:teuthology.orchestra.run.smithi066:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:10:18.935 INFO:teuthology.orchestra.run.smithi066.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:10:18.936 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:10:18.936 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:10:18.936 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:10:18.936 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:10:18.936 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T16:10:18.936 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi066.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:10:18.936 DEBUG:teuthology.orchestra.run.smithi066:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:10:19.019 INFO:teuthology.orchestra.run.smithi066.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:10:19.019 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:10:19.020 DEBUG:teuthology.orchestra.run.smithi066:> sudo rpm -qi grub2-tools 2024-03-25T16:10:19.057 INFO:teuthology.orchestra.run.smithi066.stdout:Name : grub2-tools 2024-03-25T16:10:19.057 INFO:teuthology.orchestra.run.smithi066.stdout:Epoch : 1 2024-03-25T16:10:19.058 INFO:teuthology.orchestra.run.smithi066.stdout:Version : 2.06 2024-03-25T16:10:19.058 INFO:teuthology.orchestra.run.smithi066.stdout:Release : 68.el9 2024-03-25T16:10:19.058 INFO:teuthology.orchestra.run.smithi066.stdout:Architecture: x86_64 2024-03-25T16:10:19.058 INFO:teuthology.orchestra.run.smithi066.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:10:19.058 INFO:teuthology.orchestra.run.smithi066.stdout:Group : Unspecified 2024-03-25T16:10:19.058 INFO:teuthology.orchestra.run.smithi066.stdout:Size : 8264254 2024-03-25T16:10:19.058 INFO:teuthology.orchestra.run.smithi066.stdout:License : GPLv3+ 2024-03-25T16:10:19.058 INFO:teuthology.orchestra.run.smithi066.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:10:19.058 INFO:teuthology.orchestra.run.smithi066.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:10:19.058 INFO:teuthology.orchestra.run.smithi066.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:10:19.058 INFO:teuthology.orchestra.run.smithi066.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:10:19.058 INFO:teuthology.orchestra.run.smithi066.stdout:Packager : builder@centos.org 2024-03-25T16:10:19.059 INFO:teuthology.orchestra.run.smithi066.stdout:Vendor : CentOS 2024-03-25T16:10:19.059 INFO:teuthology.orchestra.run.smithi066.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:10:19.059 INFO:teuthology.orchestra.run.smithi066.stdout:Summary : Support tools for GRUB. 2024-03-25T16:10:19.059 INFO:teuthology.orchestra.run.smithi066.stdout:Description : 2024-03-25T16:10:19.059 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:10:19.059 INFO:teuthology.orchestra.run.smithi066.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:10:19.059 INFO:teuthology.orchestra.run.smithi066.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:10:19.059 INFO:teuthology.orchestra.run.smithi066.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:10:19.059 INFO:teuthology.orchestra.run.smithi066.stdout:hardware devices. 2024-03-25T16:10:19.059 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:10:19.059 INFO:teuthology.orchestra.run.smithi066.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:10:19.060 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:10:19.061 INFO:teuthology.task.kernel:Updating grub on smithi066 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:10:19.061 DEBUG:teuthology.orchestra.run.smithi066:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:10:19.715 INFO:teuthology.orchestra.run.smithi066.stderr:Generating grub configuration file ... 2024-03-25T16:10:21.170 INFO:teuthology.orchestra.run.smithi066.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:10:21.209 INFO:teuthology.orchestra.run.smithi066.stderr:done 2024-03-25T16:10:21.211 DEBUG:teuthology.orchestra.run.smithi066:> mktemp 2024-03-25T16:10:21.226 INFO:teuthology.orchestra.run.smithi066.stdout:/tmp/tmp.08wQoRwKII 2024-03-25T16:10:21.226 DEBUG:teuthology.orchestra.run.smithi066:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.08wQoRwKII 2024-03-25T16:10:21.299 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 0666 /tmp/tmp.08wQoRwKII 2024-03-25T16:10:21.442 DEBUG:teuthology.orchestra.remote:smithi066:/tmp/tmp.08wQoRwKII is 6KB 2024-03-25T16:10:21.491 DEBUG:teuthology.orchestra.run.smithi066:> rm -fr /tmp/tmp.08wQoRwKII 2024-03-25T16:10:21.507 DEBUG:teuthology.orchestra.run.smithi066:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:10:21.575 INFO:teuthology.orchestra.run.smithi066.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:10:21.575 INFO:teuthology.orchestra.run.smithi066.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:10:21.575 INFO:teuthology.orchestra.run.smithi066.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:10:21.575 INFO:teuthology.orchestra.run.smithi066.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:10:21.576 DEBUG:teuthology.orchestra.run.smithi066:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:10:21.736 DEBUG:teuthology.orchestra.run.smithi066:> sudo shutdown -r now 2024-03-25T16:10:51.739 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T16:10:51.739 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-25T16:10:51.740 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:11:10.238 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2024-03-25T16:11:19.239 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-25T16:11:19.240 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:11:22.306 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2024-03-25T16:11:34.307 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-25T16:11:34.308 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:11:37.374 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2024-03-25T16:11:42.202 INFO:teuthology.orchestra.run.smithi193.stdout:Extra Packages for Enterprise Linux 146 kB/s | 21 MB 02:25 2024-03-25T16:11:46.976 INFO:teuthology.orchestra.run.smithi193.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-25T16:11:49.042 INFO:teuthology.orchestra.run.smithi193.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:11:49.042 INFO:teuthology.orchestra.run.smithi193.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:11:49.136 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:11:49.148 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:11:49.149 INFO:teuthology.orchestra.run.smithi193.stdout: Package Architecture Version Repository Size 2024-03-25T16:11:49.149 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:11:49.149 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T16:11:49.149 INFO:teuthology.orchestra.run.smithi193.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:11:49.149 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-25T16:11:49.149 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:11:49.149 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:11:49.149 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:11:49.149 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:11:49.149 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T16:11:49.149 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:11:49.149 INFO:teuthology.orchestra.run.smithi193.stdout:Install 4 Packages 2024-03-25T16:11:49.149 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:11:49.150 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 76 M 2024-03-25T16:11:49.150 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 123 M 2024-03-25T16:11:49.150 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T16:11:49.669 INFO:teuthology.orchestra.run.smithi193.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 185 kB/s | 44 kB 00:00 2024-03-25T16:11:51.294 INFO:teuthology.orchestra.run.smithi193.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 8.0 MB/s | 15 MB 00:01 2024-03-25T16:11:52.028 INFO:teuthology.orchestra.run.smithi193.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 13 MB/s | 33 MB 00:02 2024-03-25T16:11:52.375 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-25T16:11:52.376 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:11:52.669 INFO:teuthology.orchestra.run.smithi193.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 9.2 MB/s | 27 MB 00:02 2024-03-25T16:11:52.672 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:11:52.673 INFO:teuthology.orchestra.run.smithi193.stdout:Total 22 MB/s | 76 MB 00:03 2024-03-25T16:11:53.338 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T16:11:53.694 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T16:11:53.694 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T16:11:54.696 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T16:11:54.696 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T16:11:55.461 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.66 2024-03-25T16:11:57.170 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T16:11:57.865 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:11:57.952 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:12:00.699 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:12:00.830 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:12:05.513 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:12:05.621 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:12:10.332 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:12:13.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-25T16:12:13.463 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:12:39.696 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:12:39.696 INFO:teuthology.orchestra.run.smithi193.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:12:39.696 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:12:39.740 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:12:40.117 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:12:40.117 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:12:40.117 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:12:40.117 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:12:40.605 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:12:40.605 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:12:40.605 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T16:12:40.605 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:12:40.605 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:12:40.605 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:12:40.605 INFO:teuthology.orchestra.run.smithi193.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:12:40.605 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:12:40.605 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T16:12:40.938 DEBUG:teuthology.orchestra.run.smithi193:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:12:41.041 INFO:teuthology.orchestra.run.smithi193.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:12:41.041 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:12:41.041 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:12:41.041 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:12:41.041 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:12:41.042 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T16:12:41.042 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi193.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:12:41.042 DEBUG:teuthology.orchestra.run.smithi193:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:12:41.122 INFO:teuthology.orchestra.run.smithi193.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:12:41.122 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:12:41.122 DEBUG:teuthology.orchestra.run.smithi193:> sudo rpm -qi grub2-tools 2024-03-25T16:12:41.161 INFO:teuthology.orchestra.run.smithi193.stdout:Name : grub2-tools 2024-03-25T16:12:41.161 INFO:teuthology.orchestra.run.smithi193.stdout:Epoch : 1 2024-03-25T16:12:41.161 INFO:teuthology.orchestra.run.smithi193.stdout:Version : 2.06 2024-03-25T16:12:41.161 INFO:teuthology.orchestra.run.smithi193.stdout:Release : 68.el9 2024-03-25T16:12:41.161 INFO:teuthology.orchestra.run.smithi193.stdout:Architecture: x86_64 2024-03-25T16:12:41.161 INFO:teuthology.orchestra.run.smithi193.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:12:41.161 INFO:teuthology.orchestra.run.smithi193.stdout:Group : Unspecified 2024-03-25T16:12:41.161 INFO:teuthology.orchestra.run.smithi193.stdout:Size : 8264254 2024-03-25T16:12:41.161 INFO:teuthology.orchestra.run.smithi193.stdout:License : GPLv3+ 2024-03-25T16:12:41.161 INFO:teuthology.orchestra.run.smithi193.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:12:41.161 INFO:teuthology.orchestra.run.smithi193.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:12:41.162 INFO:teuthology.orchestra.run.smithi193.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:12:41.162 INFO:teuthology.orchestra.run.smithi193.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:12:41.162 INFO:teuthology.orchestra.run.smithi193.stdout:Packager : builder@centos.org 2024-03-25T16:12:41.162 INFO:teuthology.orchestra.run.smithi193.stdout:Vendor : CentOS 2024-03-25T16:12:41.162 INFO:teuthology.orchestra.run.smithi193.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:12:41.162 INFO:teuthology.orchestra.run.smithi193.stdout:Summary : Support tools for GRUB. 2024-03-25T16:12:41.162 INFO:teuthology.orchestra.run.smithi193.stdout:Description : 2024-03-25T16:12:41.162 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:12:41.162 INFO:teuthology.orchestra.run.smithi193.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:12:41.162 INFO:teuthology.orchestra.run.smithi193.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:12:41.162 INFO:teuthology.orchestra.run.smithi193.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:12:41.162 INFO:teuthology.orchestra.run.smithi193.stdout:hardware devices. 2024-03-25T16:12:41.162 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:12:41.162 INFO:teuthology.orchestra.run.smithi193.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:12:41.163 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:12:41.163 INFO:teuthology.task.kernel:Updating grub on smithi193 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:12:41.164 DEBUG:teuthology.orchestra.run.smithi193:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:12:41.769 INFO:teuthology.orchestra.run.smithi193.stderr:Generating grub configuration file ... 2024-03-25T16:12:43.212 INFO:teuthology.orchestra.run.smithi193.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:12:43.246 INFO:teuthology.orchestra.run.smithi193.stderr:done 2024-03-25T16:12:43.248 DEBUG:teuthology.orchestra.run.smithi193:> mktemp 2024-03-25T16:12:43.263 INFO:teuthology.orchestra.run.smithi193.stdout:/tmp/tmp.T9k81V4vYr 2024-03-25T16:12:43.263 DEBUG:teuthology.orchestra.run.smithi193:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.T9k81V4vYr 2024-03-25T16:12:43.346 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 0666 /tmp/tmp.T9k81V4vYr 2024-03-25T16:12:43.489 DEBUG:teuthology.orchestra.remote:smithi193:/tmp/tmp.T9k81V4vYr is 6KB 2024-03-25T16:12:43.538 DEBUG:teuthology.orchestra.run.smithi193:> rm -fr /tmp/tmp.T9k81V4vYr 2024-03-25T16:12:43.551 DEBUG:teuthology.orchestra.run.smithi193:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:12:43.623 INFO:teuthology.orchestra.run.smithi193.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:12:43.623 INFO:teuthology.orchestra.run.smithi193.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:12:43.623 INFO:teuthology.orchestra.run.smithi193.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:12:43.623 INFO:teuthology.orchestra.run.smithi193.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:12:43.624 DEBUG:teuthology.orchestra.run.smithi193:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:12:43.758 DEBUG:teuthology.orchestra.run.smithi193:> sudo shutdown -r now 2024-03-25T16:13:13.495 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T16:13:13.762 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T16:13:13.762 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-25T16:13:13.763 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:13:32.318 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.193 2024-03-25T16:13:34.498 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-25T16:13:34.505 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:13:34.943 DEBUG:teuthology.orchestra.run.smithi066:> true 2024-03-25T16:13:35.326 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-25T16:13:35.326 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:13:35.327 DEBUG:teuthology.orchestra.run.smithi066:> uname -r 2024-03-25T16:13:35.343 INFO:teuthology.orchestra.run.smithi066.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:13:35.343 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:13:35.343 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:13:35.343 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:13:36.344 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T16:13:36.345 DEBUG:teuthology.orchestra.run.smithi066:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:13:36.430 INFO:teuthology.orchestra.run.smithi066.stdout:ttyS1 2024-03-25T16:13:36.467 DEBUG:teuthology.parallel:result is None 2024-03-25T16:13:41.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-25T16:13:41.320 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:14:41.322 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T16:14:53.324 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-25T16:14:53.326 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:14:53.679 DEBUG:teuthology.orchestra.run.smithi193:> true 2024-03-25T16:14:54.055 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-25T16:14:54.055 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:14:54.055 DEBUG:teuthology.orchestra.run.smithi193:> uname -r 2024-03-25T16:14:54.071 INFO:teuthology.orchestra.run.smithi193.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:14:54.071 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:14:54.071 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:14:54.071 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:14:55.072 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T16:14:55.073 DEBUG:teuthology.orchestra.run.smithi193:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:14:55.155 INFO:teuthology.orchestra.run.smithi193.stdout:ttyS1 2024-03-25T16:14:55.187 DEBUG:teuthology.parallel:result is None 2024-03-25T16:14:55.187 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T16:14:55.227 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T16:14:55.228 DEBUG:teuthology.orchestra.run.smithi066:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:14:55.230 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:14:55.249 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T16:14:55.255 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T16:14:55.262 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T16:14:55.262 DEBUG:teuthology.orchestra.run.smithi066:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:14:55.290 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:14:55.339 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T16:14:55.346 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T16:14:55.347 DEBUG:teuthology.orchestra.run.smithi066:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:14:55.357 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:14:55.396 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:14:55.402 INFO:teuthology.orchestra.run.smithi066.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:14:55.411 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:14:55.420 INFO:teuthology.orchestra.run.smithi066.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:14:55.421 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T16:14:55.471 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T16:14:55.471 DEBUG:teuthology.orchestra.run.smithi066:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:14:55.473 DEBUG:teuthology.orchestra.run.smithi193:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:14:55.505 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T16:14:55.514 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T16:14:55.514 DEBUG:teuthology.orchestra.run.smithi066:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:14:55.548 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:14:55.564 DEBUG:teuthology.orchestra.run.smithi066:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:14:55.646 DEBUG:teuthology.orchestra.run.smithi066:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:14:55.697 DEBUG:teuthology.orchestra.run.smithi066:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:14:55.780 DEBUG:teuthology.orchestra.run.smithi066:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:14:55.827 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-25T16:14:55.827 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:14:55.899 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:14:55.943 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:14:55.996 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:14:56.080 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:14:56.124 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T16:14:56.125 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:14:56.193 DEBUG:teuthology.orchestra.run.smithi066:> sudo service rsyslog restart 2024-03-25T16:14:56.196 DEBUG:teuthology.orchestra.run.smithi193:> sudo service rsyslog restart 2024-03-25T16:14:56.243 INFO:teuthology.orchestra.run.smithi066.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:14:56.275 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:14:56.613 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T16:14:56.623 INFO:teuthology.task.internal:Starting timer... 2024-03-25T16:14:56.623 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T16:14:56.640 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T16:14:56.648 DEBUG:teuthology.orchestra.run.smithi066:> sudo service auditd rotate 2024-03-25T16:14:56.703 INFO:teuthology.orchestra.run.smithi066.stdout:Rotating logs: 2024-03-25T16:14:56.704 DEBUG:teuthology.orchestra.run.smithi193:> sudo service auditd rotate 2024-03-25T16:14:56.766 INFO:teuthology.orchestra.run.smithi193.stdout:Rotating logs: 2024-03-25T16:14:56.767 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T16:14:56.768 DEBUG:teuthology.orchestra.run.smithi066:> /usr/sbin/getenforce 2024-03-25T16:14:56.793 INFO:teuthology.orchestra.run.smithi066.stdout:Permissive 2024-03-25T16:14:56.793 DEBUG:teuthology.orchestra.run.smithi193:> /usr/sbin/getenforce 2024-03-25T16:14:56.818 INFO:teuthology.orchestra.run.smithi193.stdout:Permissive 2024-03-25T16:14:56.818 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi066.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi193.front.sepia.ceph.com': 'permissive'} 2024-03-25T16:14:56.818 DEBUG:teuthology.orchestra.run.smithi066:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:14:56.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:14:56.866 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:14:56.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:14:56.896 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T16:14:56.896 DEBUG:teuthology.orchestra.run.smithi066:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:14:56.945 DEBUG:teuthology.orchestra.run.smithi193:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:14:56.977 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T16:14:56.987 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T16:14:57.005 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T16:14:57.007 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi066.front.sepia.ceph.com,smithi193.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T16:19:15.971 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi066.front.sepia.ceph.com'), Remote(name='ubuntu@smithi193.front.sepia.ceph.com')] 2024-03-25T16:19:15.973 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-25T16:19:15.974 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi066.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:19:16.084 DEBUG:teuthology.orchestra.run.smithi066:> true 2024-03-25T16:19:16.169 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi066.front.sepia.ceph.com' 2024-03-25T16:19:16.169 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-25T16:19:16.170 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:19:16.259 DEBUG:teuthology.orchestra.run.smithi193:> true 2024-03-25T16:19:16.343 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-25T16:19:16.343 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T16:19:16.354 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T16:19:16.354 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:19:16.355 DEBUG:teuthology.orchestra.run.smithi066:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:19:16.357 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:19:16.357 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:19:16.390 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:19:16.411 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:19:16.416 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:19:16.433 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:19:16.447 INFO:teuthology.orchestra.run.smithi066.stderr:sudo: ntpd: command not found 2024-03-25T16:19:16.463 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: ntpd: command not found 2024-03-25T16:19:16.464 INFO:teuthology.orchestra.run.smithi066.stdout:506 Cannot talk to daemon 2024-03-25T16:19:16.477 INFO:teuthology.orchestra.run.smithi193.stdout:506 Cannot talk to daemon 2024-03-25T16:19:16.484 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:19:16.493 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:19:16.504 INFO:teuthology.orchestra.run.smithi066.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:19:16.509 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:19:16.560 INFO:teuthology.orchestra.run.smithi066.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:19:16.560 INFO:teuthology.orchestra.run.smithi193.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:19:16.563 INFO:teuthology.orchestra.run.smithi066.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:19:16.563 INFO:teuthology.orchestra.run.smithi066.stdout:=============================================================================== 2024-03-25T16:19:16.563 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:19:16.563 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:19:16.563 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:19:16.564 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:19:16.565 INFO:teuthology.orchestra.run.smithi193.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:19:16.565 INFO:teuthology.orchestra.run.smithi193.stdout:=============================================================================== 2024-03-25T16:19:16.565 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:19:16.565 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:19:16.565 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:19:16.565 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:19:16.566 INFO:teuthology.run_tasks:Running task install... 2024-03-25T16:19:16.574 DEBUG:teuthology.task.install:project ceph 2024-03-25T16:19:16.574 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T16:19:16.574 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T16:19:16.575 INFO:teuthology.task.install:Using flavor: default 2024-03-25T16:19:16.585 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T16:19:16.585 INFO:teuthology.task.install:extra packages: [] 2024-03-25T16:19:16.586 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:19:16.586 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:19:16.589 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:19:16.589 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:19:16.745 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:19:16.745 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:19:16.805 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:19:16.805 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:19:17.068 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:19:17.068 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-25T16:19:17.068 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:19:17.102 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:19:17.102 DEBUG:teuthology.orchestra.run.smithi066:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:19:17.130 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:19:17.130 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T16:19:17.130 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:19:17.159 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:19:17.160 DEBUG:teuthology.orchestra.run.smithi193:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:19:17.174 DEBUG:teuthology.orchestra.run.smithi066:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:19:17.228 DEBUG:teuthology.orchestra.run.smithi193:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:19:17.263 DEBUG:teuthology.orchestra.run.smithi066:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:19:17.293 INFO:teuthology.orchestra.run.smithi066.stdout:check_obsoletes = 1 2024-03-25T16:19:17.294 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum clean all 2024-03-25T16:19:17.308 DEBUG:teuthology.orchestra.run.smithi193:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:19:17.375 INFO:teuthology.orchestra.run.smithi193.stdout:check_obsoletes = 1 2024-03-25T16:19:17.376 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum clean all 2024-03-25T16:19:17.551 INFO:teuthology.orchestra.run.smithi066.stdout:74 files removed 2024-03-25T16:19:17.579 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-radosgw 2024-03-25T16:19:17.624 INFO:teuthology.orchestra.run.smithi193.stdout:74 files removed 2024-03-25T16:19:17.651 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-radosgw 2024-03-25T16:19:18.979 INFO:teuthology.orchestra.run.smithi066.stdout:ceph packages for x86_64 76 kB/s | 82 kB 00:01 2024-03-25T16:19:19.066 INFO:teuthology.orchestra.run.smithi193.stdout:ceph packages for x86_64 73 kB/s | 82 kB 00:01 2024-03-25T16:19:19.863 INFO:teuthology.orchestra.run.smithi066.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T16:19:19.942 INFO:teuthology.orchestra.run.smithi193.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T16:19:20.764 INFO:teuthology.orchestra.run.smithi066.stdout:ceph source packages 2.2 kB/s | 1.9 kB 00:00 2024-03-25T16:19:20.826 INFO:teuthology.orchestra.run.smithi193.stdout:ceph source packages 2.2 kB/s | 1.9 kB 00:00 2024-03-25T16:19:21.199 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-25T16:19:21.369 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-25T16:19:22.633 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-03-25T16:19:22.944 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS-9 - BaseOS 4.2 MB/s | 2.0 MB 00:00 2024-03-25T16:19:23.440 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-03-25T16:19:23.852 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-03-25T16:19:45.843 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - AppStream 921 kB/s | 19 MB 00:20 2024-03-25T16:19:45.906 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - AppStream 902 kB/s | 19 MB 00:21 2024-03-25T16:19:49.784 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-25T16:19:50.532 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - CRB 6.1 MB/s | 6.1 MB 00:01 2024-03-25T16:19:51.586 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-03-25T16:19:52.233 INFO:teuthology.orchestra.run.smithi066.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 16 kB 00:00 2024-03-25T16:19:53.228 INFO:teuthology.orchestra.run.smithi193.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-25T16:19:53.501 INFO:teuthology.orchestra.run.smithi066.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-03-25T16:19:58.245 INFO:teuthology.orchestra.run.smithi193.stdout:lab-extras 8.3 kB/s | 1.7 kB 00:00 2024-03-25T16:19:58.835 INFO:teuthology.orchestra.run.smithi066.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-03-25T16:20:00.682 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:20:00.690 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:20:00.690 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-25T16:20:00.690 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:20:00.693 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T16:20:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:20:00.693 INFO:teuthology.orchestra.run.smithi193.stdout:Upgrading: 2024-03-25T16:20:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:20:00.693 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:20:00.693 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-25T16:20:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: boost-program-options 2024-03-25T16:20:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:20:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:20:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:20:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:20:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:20:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:20:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:20:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:20:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:20:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:20:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:20:00.694 INFO:teuthology.orchestra.run.smithi193.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:20:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:20:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:20:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:20:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:20:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:20:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-argparse 2024-03-25T16:20:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:20:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:20:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:20:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:20:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:20:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:20:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:20:00.695 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:20:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:20:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:20:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:20:00.696 INFO:teuthology.orchestra.run.smithi193.stdout:Installing weak dependencies: 2024-03-25T16:20:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:20:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:20:00.696 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T16:20:00.696 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:20:00.696 INFO:teuthology.orchestra.run.smithi193.stdout:Install 30 Packages 2024-03-25T16:20:00.696 INFO:teuthology.orchestra.run.smithi193.stdout:Upgrade 2 Packages 2024-03-25T16:20:00.696 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:20:00.697 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 58 M 2024-03-25T16:20:00.697 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T16:20:01.384 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:20:01.401 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:20:01.401 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-03-25T16:20:01.401 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:20:01.401 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T16:20:01.401 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:20:01.401 INFO:teuthology.orchestra.run.smithi066.stdout:Upgrading: 2024-03-25T16:20:01.401 INFO:teuthology.orchestra.run.smithi066.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:20:01.401 INFO:teuthology.orchestra.run.smithi066.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:20:01.402 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-25T16:20:01.402 INFO:teuthology.orchestra.run.smithi066.stdout: boost-program-options 2024-03-25T16:20:01.402 INFO:teuthology.orchestra.run.smithi066.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:20:01.402 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:20:01.402 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:20:01.402 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:20:01.402 INFO:teuthology.orchestra.run.smithi066.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:20:01.402 INFO:teuthology.orchestra.run.smithi066.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:20:01.402 INFO:teuthology.orchestra.run.smithi066.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:20:01.402 INFO:teuthology.orchestra.run.smithi066.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:20:01.402 INFO:teuthology.orchestra.run.smithi066.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:20:01.402 INFO:teuthology.orchestra.run.smithi066.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:20:01.403 INFO:teuthology.orchestra.run.smithi066.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:20:01.403 INFO:teuthology.orchestra.run.smithi066.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:20:01.403 INFO:teuthology.orchestra.run.smithi066.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:20:01.403 INFO:teuthology.orchestra.run.smithi066.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:20:01.403 INFO:teuthology.orchestra.run.smithi066.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:20:01.403 INFO:teuthology.orchestra.run.smithi066.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:20:01.403 INFO:teuthology.orchestra.run.smithi066.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:20:01.403 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ceph-argparse 2024-03-25T16:20:01.403 INFO:teuthology.orchestra.run.smithi066.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:20:01.403 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:20:01.403 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:20:01.403 INFO:teuthology.orchestra.run.smithi066.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:20:01.403 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:20:01.404 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:20:01.404 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:20:01.404 INFO:teuthology.orchestra.run.smithi066.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:20:01.404 INFO:teuthology.orchestra.run.smithi066.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:20:01.404 INFO:teuthology.orchestra.run.smithi066.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:20:01.404 INFO:teuthology.orchestra.run.smithi066.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:20:01.404 INFO:teuthology.orchestra.run.smithi066.stdout:Installing weak dependencies: 2024-03-25T16:20:01.404 INFO:teuthology.orchestra.run.smithi066.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:20:01.404 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:20:01.404 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T16:20:01.404 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:20:01.404 INFO:teuthology.orchestra.run.smithi066.stdout:Install 30 Packages 2024-03-25T16:20:01.404 INFO:teuthology.orchestra.run.smithi066.stdout:Upgrade 2 Packages 2024-03-25T16:20:01.404 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:20:01.405 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 58 M 2024-03-25T16:20:01.405 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T16:20:07.687 INFO:teuthology.orchestra.run.smithi193.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 819 kB/s | 5.3 MB 00:06 2024-03-25T16:20:07.795 INFO:teuthology.orchestra.run.smithi193.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 232 kB/s | 25 kB 00:00 2024-03-25T16:20:08.603 INFO:teuthology.orchestra.run.smithi193.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 918 kB/s | 742 kB 00:00 2024-03-25T16:20:09.095 INFO:teuthology.orchestra.run.smithi193.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 1.0 MB/s | 498 kB 00:00 2024-03-25T16:20:10.185 INFO:teuthology.orchestra.run.smithi066.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 630 kB/s | 5.3 MB 00:08 2024-03-25T16:20:10.302 INFO:teuthology.orchestra.run.smithi066.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 216 kB/s | 25 kB 00:00 2024-03-25T16:20:11.477 INFO:teuthology.orchestra.run.smithi066.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 631 kB/s | 742 kB 00:01 2024-03-25T16:20:12.235 INFO:teuthology.orchestra.run.smithi066.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 657 kB/s | 498 kB 00:00 2024-03-25T16:20:12.637 INFO:teuthology.orchestra.run.smithi193.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 1.4 MB/s | 5.1 MB 00:03 2024-03-25T16:20:12.729 INFO:teuthology.orchestra.run.smithi193.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 494 kB/s | 45 kB 00:00 2024-03-25T16:20:12.821 INFO:teuthology.orchestra.run.smithi193.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.4 MB/s | 131 kB 00:00 2024-03-25T16:20:13.054 INFO:teuthology.orchestra.run.smithi193.stdout:(8/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 828 kB/s | 9.7 MB 00:12 2024-03-25T16:20:13.079 INFO:teuthology.orchestra.run.smithi193.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 626 kB/s | 162 kB 00:00 2024-03-25T16:20:13.247 INFO:teuthology.orchestra.run.smithi193.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 1.7 MB/s | 298 kB 00:00 2024-03-25T16:20:13.347 INFO:teuthology.orchestra.run.smithi193.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 995 kB/s | 99 kB 00:00 2024-03-25T16:20:13.363 INFO:teuthology.orchestra.run.smithi066.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 840 kB/s | 9.7 MB 00:11 2024-03-25T16:20:13.455 INFO:teuthology.orchestra.run.smithi193.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 977 kB/s | 106 kB 00:00 2024-03-25T16:20:13.461 INFO:teuthology.orchestra.run.smithi066.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 463 kB/s | 45 kB 00:00 2024-03-25T16:20:13.498 INFO:teuthology.orchestra.run.smithi193.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-25T16:20:13.531 INFO:teuthology.orchestra.run.smithi193.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T16:20:13.590 INFO:teuthology.orchestra.run.smithi193.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T16:20:13.628 INFO:teuthology.orchestra.run.smithi066.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 787 kB/s | 131 kB 00:00 2024-03-25T16:20:13.631 INFO:teuthology.orchestra.run.smithi193.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T16:20:13.666 INFO:teuthology.orchestra.run.smithi193.stdout:(17/32): python3-rados-19.0.0-1439.g5dcb190d.el 525 kB/s | 320 kB 00:00 2024-03-25T16:20:13.690 INFO:teuthology.orchestra.run.smithi193.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 702 kB/s | 41 kB 00:00 2024-03-25T16:20:13.816 INFO:teuthology.orchestra.run.smithi193.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 279 kB/s | 35 kB 00:00 2024-03-25T16:20:13.870 INFO:teuthology.orchestra.run.smithi066.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 670 kB/s | 162 kB 00:00 2024-03-25T16:20:13.874 INFO:teuthology.orchestra.run.smithi193.stdout:(20/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.0 MB/s | 216 kB 00:00 2024-03-25T16:20:13.899 INFO:teuthology.orchestra.run.smithi193.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 532 kB/s | 44 kB 00:00 2024-03-25T16:20:14.250 INFO:teuthology.orchestra.run.smithi193.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 823 kB/s | 309 kB 00:00 2024-03-25T16:20:14.278 INFO:teuthology.orchestra.run.smithi066.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 784 kB/s | 320 kB 00:00 2024-03-25T16:20:14.316 INFO:teuthology.orchestra.run.smithi193.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 389 kB/s | 26 kB 00:00 2024-03-25T16:20:14.384 INFO:teuthology.orchestra.run.smithi193.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 729 kB/s | 49 kB 00:00 2024-03-25T16:20:14.467 INFO:teuthology.orchestra.run.smithi193.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 810 kB/s | 67 kB 00:00 2024-03-25T16:20:14.526 INFO:teuthology.orchestra.run.smithi193.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 7.1 MB/s | 4.4 MB 00:00 2024-03-25T16:20:14.593 INFO:teuthology.orchestra.run.smithi193.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-25T16:20:14.635 INFO:teuthology.orchestra.run.smithi193.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.9 MB/s | 838 kB 00:00 2024-03-25T16:20:14.693 INFO:teuthology.orchestra.run.smithi193.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-03-25T16:20:14.695 INFO:teuthology.orchestra.run.smithi066.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 715 kB/s | 298 kB 00:00 2024-03-25T16:20:14.963 INFO:teuthology.orchestra.run.smithi066.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 372 kB/s | 99 kB 00:00 2024-03-25T16:20:15.079 INFO:teuthology.orchestra.run.smithi066.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 911 kB/s | 106 kB 00:00 2024-03-25T16:20:15.130 INFO:teuthology.orchestra.run.smithi066.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-25T16:20:15.163 INFO:teuthology.orchestra.run.smithi066.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T16:20:15.222 INFO:teuthology.orchestra.run.smithi066.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T16:20:15.255 INFO:teuthology.orchestra.run.smithi066.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 8.3 MB/s | 282 kB 00:00 2024-03-25T16:20:15.289 INFO:teuthology.orchestra.run.smithi066.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T16:20:15.448 INFO:teuthology.orchestra.run.smithi066.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.3 MB/s | 216 kB 00:00 2024-03-25T16:20:15.498 INFO:teuthology.orchestra.run.smithi066.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 697 kB/s | 35 kB 00:00 2024-03-25T16:20:15.548 INFO:teuthology.orchestra.run.smithi066.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 896 kB/s | 44 kB 00:00 2024-03-25T16:20:16.565 INFO:teuthology.orchestra.run.smithi066.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 304 kB/s | 309 kB 00:01 2024-03-25T16:20:19.714 INFO:teuthology.orchestra.run.smithi193.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 639 kB/s | 3.1 MB 00:05 2024-03-25T16:20:19.957 INFO:teuthology.orchestra.run.smithi066.stdout:(22/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 676 kB/s | 5.1 MB 00:07 2024-03-25T16:20:20.024 INFO:teuthology.orchestra.run.smithi066.stdout:(23/32): libarrow-9.0.0-10.el9.x86_64.rpm 1.3 MB/s | 4.4 MB 00:03 2024-03-25T16:20:20.027 INFO:teuthology.orchestra.run.smithi193.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 641 kB/s | 3.4 MB 00:05 2024-03-25T16:20:20.149 INFO:teuthology.orchestra.run.smithi066.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 389 kB/s | 49 kB 00:00 2024-03-25T16:20:20.258 INFO:teuthology.orchestra.run.smithi066.stdout:(25/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 86 kB/s | 26 kB 00:00 2024-03-25T16:20:20.283 INFO:teuthology.orchestra.run.smithi066.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 504 kB/s | 67 kB 00:00 2024-03-25T16:20:20.400 INFO:teuthology.orchestra.run.smithi066.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-03-25T16:20:20.975 INFO:teuthology.orchestra.run.smithi066.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 2.8 MB/s | 1.6 MB 00:00 2024-03-25T16:20:21.950 INFO:teuthology.orchestra.run.smithi066.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 495 kB/s | 838 kB 00:01 2024-03-25T16:20:22.467 INFO:teuthology.orchestra.run.smithi066.stdout:(30/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 971 kB/s | 20 MB 00:20 2024-03-25T16:20:26.634 INFO:teuthology.orchestra.run.smithi066.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 684 kB/s | 3.1 MB 00:04 2024-03-25T16:20:26.868 INFO:teuthology.orchestra.run.smithi193.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 787 kB/s | 20 MB 00:25 2024-03-25T16:20:26.873 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:20:26.873 INFO:teuthology.orchestra.run.smithi193.stdout:Total 2.2 MB/s | 58 MB 00:26 2024-03-25T16:20:26.903 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T16:20:26.939 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T16:20:26.939 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T16:20:27.417 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T16:20:27.417 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T16:20:27.600 INFO:teuthology.orchestra.run.smithi066.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 521 kB/s | 3.4 MB 00:06 2024-03-25T16:20:27.604 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:20:27.604 INFO:teuthology.orchestra.run.smithi066.stdout:Total 2.2 MB/s | 58 MB 00:26 2024-03-25T16:20:27.638 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T16:20:27.680 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T16:20:27.681 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T16:20:28.169 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T16:20:28.170 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T16:20:28.911 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T16:20:29.162 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:20:29.265 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:20:29.290 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:20:29.313 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:20:29.331 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:20:29.528 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:20:29.587 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:20:29.767 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T16:20:29.916 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:20:29.942 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:20:29.983 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:20:30.084 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:20:30.140 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:20:30.157 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:20:30.169 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:20:30.179 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:20:30.192 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:20:30.198 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:20:30.238 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:20:30.267 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:20:30.310 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:20:30.378 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:20:30.400 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:20:30.403 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:20:30.462 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:20:30.466 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:20:30.493 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:20:30.599 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:20:30.636 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:20:30.689 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:20:30.717 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:20:30.746 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:20:30.778 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:20:30.803 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:20:30.849 INFO:teuthology.orchestra.run.smithi066.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:20:30.934 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:20:31.022 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:20:31.050 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:20:31.095 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:20:31.096 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:20:31.125 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:20:31.125 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:20:31.169 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:20:31.206 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:20:31.239 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:20:31.239 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:20:31.261 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:20:31.281 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:20:31.324 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:20:31.351 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:20:31.463 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:20:31.494 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:20:31.534 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:20:31.567 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:20:31.596 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:20:31.629 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:20:31.730 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:20:31.761 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:20:31.832 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:20:31.901 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:20:31.955 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:20:31.962 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:20:32.008 INFO:teuthology.orchestra.run.smithi066.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:20:32.091 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:20:32.122 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:20:32.169 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:20:32.645 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:20:32.688 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:20:32.759 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:20:32.831 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:20:32.865 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:20:34.135 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:20:34.177 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:20:34.720 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:20:34.763 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:20:34.852 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:20:34.852 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:20:34.852 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:20:34.853 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:20:34.881 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:20:35.096 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:20:35.136 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:20:35.718 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:20:35.756 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:20:35.839 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:20:35.841 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:20:35.841 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:20:35.841 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:20:35.874 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:20:44.049 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:20:44.049 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /sys 2024-03-25T16:20:44.049 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /proc 2024-03-25T16:20:44.049 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /mnt 2024-03-25T16:20:44.049 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /var/tmp 2024-03-25T16:20:44.049 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /home 2024-03-25T16:20:44.049 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /root 2024-03-25T16:20:44.049 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /tmp 2024-03-25T16:20:44.049 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:20:44.961 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:20:44.986 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:20:44.986 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:20:44.986 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:20:44.986 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:20:44.986 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:20:44.986 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:20:44.989 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:20:45.073 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:20:45.073 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:20:45.519 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:20:45.519 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /sys 2024-03-25T16:20:45.519 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /proc 2024-03-25T16:20:45.520 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /mnt 2024-03-25T16:20:45.520 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /var/tmp 2024-03-25T16:20:45.520 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /home 2024-03-25T16:20:45.520 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /root 2024-03-25T16:20:45.520 INFO:teuthology.orchestra.run.smithi066.stdout:skipping the directory /tmp 2024-03-25T16:20:45.520 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:20:46.143 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:20:46.143 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:20:46.143 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:20:46.143 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:20:46.143 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:20:46.143 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:20:46.144 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:20:46.144 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:20:46.144 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:20:46.144 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:20:46.144 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:20:46.144 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:20:46.144 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:20:46.144 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:20:46.144 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:20:46.144 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:20:46.144 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:20:46.144 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:20:46.144 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:20:46.144 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:20:46.145 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:20:46.145 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:20:46.145 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:20:46.145 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:20:46.145 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:20:46.145 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:20:46.145 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:20:46.145 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:20:46.145 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:20:46.145 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:20:46.145 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:20:46.146 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:20:46.146 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:20:46.146 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:20:46.449 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:20:46.475 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:20:46.475 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:20:46.475 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:20:46.475 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:20:46.475 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:20:46.475 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:20:46.477 INFO:teuthology.orchestra.run.smithi066.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:20:46.560 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:20:46.561 INFO:teuthology.orchestra.run.smithi066.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:20:47.297 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:20:47.297 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:20:47.297 INFO:teuthology.orchestra.run.smithi193.stdout:Upgraded: 2024-03-25T16:20:47.297 INFO:teuthology.orchestra.run.smithi193.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:47.297 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:47.298 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T16:20:47.298 INFO:teuthology.orchestra.run.smithi193.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:20:47.298 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:47.298 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:47.299 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:47.299 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:47.299 INFO:teuthology.orchestra.run.smithi193.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:20:47.299 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:20:47.299 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:20:47.299 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:47.299 INFO:teuthology.orchestra.run.smithi193.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:20:47.300 INFO:teuthology.orchestra.run.smithi193.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:20:47.300 INFO:teuthology.orchestra.run.smithi193.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:20:47.300 INFO:teuthology.orchestra.run.smithi193.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:47.300 INFO:teuthology.orchestra.run.smithi193.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:20:47.300 INFO:teuthology.orchestra.run.smithi193.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:47.300 INFO:teuthology.orchestra.run.smithi193.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:20:47.301 INFO:teuthology.orchestra.run.smithi193.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:20:47.301 INFO:teuthology.orchestra.run.smithi193.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:20:47.301 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:47.301 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:47.301 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:47.301 INFO:teuthology.orchestra.run.smithi193.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:20:47.301 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:47.302 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:47.302 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:47.302 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:20:47.302 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:20:47.302 INFO:teuthology.orchestra.run.smithi193.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:20:47.302 INFO:teuthology.orchestra.run.smithi193.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:20:47.303 INFO:teuthology.orchestra.run.smithi193.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:20:47.303 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:20:47.303 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T16:20:47.656 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-test 2024-03-25T16:20:47.702 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:20:47.703 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:20:47.703 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:20:47.703 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:20:47.703 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:20:47.703 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:20:47.703 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:20:47.703 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:20:47.703 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:20:47.703 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:20:47.703 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:20:47.703 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:20:47.703 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:20:47.704 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:20:47.704 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:20:47.704 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:20:47.704 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:20:47.704 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:20:47.704 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:20:47.704 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:20:47.704 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:20:47.704 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:20:47.704 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:20:47.704 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:20:47.704 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:20:47.704 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:20:47.704 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:20:47.705 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:20:47.705 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:20:47.705 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:20:47.705 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:20:47.705 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:20:47.705 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:20:47.705 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:20:48.218 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:00:50 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:20:48.368 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:20:48.368 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:20:48.368 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-25T16:20:48.368 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:20:48.369 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T16:20:48.369 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:20:48.369 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-25T16:20:48.369 INFO:teuthology.orchestra.run.smithi193.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:20:48.369 INFO:teuthology.orchestra.run.smithi193.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:20:48.369 INFO:teuthology.orchestra.run.smithi193.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:20:48.369 INFO:teuthology.orchestra.run.smithi193.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:20:48.369 INFO:teuthology.orchestra.run.smithi193.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:20:48.369 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:20:48.369 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T16:20:48.369 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:20:48.369 INFO:teuthology.orchestra.run.smithi193.stdout:Install 6 Packages 2024-03-25T16:20:48.369 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:20:48.370 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 48 M 2024-03-25T16:20:48.370 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 204 M 2024-03-25T16:20:48.370 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T16:20:48.645 INFO:teuthology.orchestra.run.smithi193.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-25T16:20:48.696 INFO:teuthology.orchestra.run.smithi193.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-25T16:20:48.737 INFO:teuthology.orchestra.run.smithi193.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-25T16:20:48.871 INFO:teuthology.orchestra.run.smithi193.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-25T16:20:48.924 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:20:48.924 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:20:48.924 INFO:teuthology.orchestra.run.smithi066.stdout:Upgraded: 2024-03-25T16:20:48.924 INFO:teuthology.orchestra.run.smithi066.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:48.925 INFO:teuthology.orchestra.run.smithi066.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:48.925 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T16:20:48.925 INFO:teuthology.orchestra.run.smithi066.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:20:48.925 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:48.925 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:48.925 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:48.925 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:48.925 INFO:teuthology.orchestra.run.smithi066.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:20:48.925 INFO:teuthology.orchestra.run.smithi066.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:20:48.925 INFO:teuthology.orchestra.run.smithi066.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:20:48.925 INFO:teuthology.orchestra.run.smithi066.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:48.925 INFO:teuthology.orchestra.run.smithi066.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:20:48.925 INFO:teuthology.orchestra.run.smithi066.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:20:48.925 INFO:teuthology.orchestra.run.smithi066.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:20:48.926 INFO:teuthology.orchestra.run.smithi066.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:48.926 INFO:teuthology.orchestra.run.smithi066.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:20:48.926 INFO:teuthology.orchestra.run.smithi066.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:48.926 INFO:teuthology.orchestra.run.smithi066.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:20:48.926 INFO:teuthology.orchestra.run.smithi066.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:20:48.926 INFO:teuthology.orchestra.run.smithi066.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:20:48.926 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:48.926 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:48.926 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:48.926 INFO:teuthology.orchestra.run.smithi066.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:20:48.926 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:48.926 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:48.926 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:48.926 INFO:teuthology.orchestra.run.smithi066.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:20:48.927 INFO:teuthology.orchestra.run.smithi066.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:20:48.927 INFO:teuthology.orchestra.run.smithi066.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:20:48.927 INFO:teuthology.orchestra.run.smithi066.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:20:48.927 INFO:teuthology.orchestra.run.smithi066.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:20:48.927 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:20:48.927 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T16:20:49.271 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-test 2024-03-25T16:20:49.388 INFO:teuthology.orchestra.run.smithi193.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 182 kB/s | 163 kB 00:00 2024-03-25T16:20:49.860 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:00:51 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:20:50.013 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:20:50.014 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:20:50.014 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-03-25T16:20:50.014 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:20:50.014 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T16:20:50.014 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:20:50.014 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-25T16:20:50.014 INFO:teuthology.orchestra.run.smithi066.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:20:50.015 INFO:teuthology.orchestra.run.smithi066.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:20:50.015 INFO:teuthology.orchestra.run.smithi066.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:20:50.015 INFO:teuthology.orchestra.run.smithi066.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:20:50.015 INFO:teuthology.orchestra.run.smithi066.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:20:50.015 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:20:50.015 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T16:20:50.015 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:20:50.015 INFO:teuthology.orchestra.run.smithi066.stdout:Install 6 Packages 2024-03-25T16:20:50.015 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:20:50.016 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 48 M 2024-03-25T16:20:50.016 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 204 M 2024-03-25T16:20:50.016 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T16:20:50.422 INFO:teuthology.orchestra.run.smithi066.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.6 MB/s | 217 kB 00:00 2024-03-25T16:20:50.489 INFO:teuthology.orchestra.run.smithi066.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 6.8 MB/s | 300 kB 00:00 2024-03-25T16:20:50.514 INFO:teuthology.orchestra.run.smithi066.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-03-25T16:20:50.739 INFO:teuthology.orchestra.run.smithi066.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 831 kB/s | 187 kB 00:00 2024-03-25T16:20:50.906 INFO:teuthology.orchestra.run.smithi066.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 266 kB/s | 163 kB 00:00 2024-03-25T16:21:23.467 INFO:teuthology.orchestra.run.smithi193.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 1.4 MB/s | 47 MB 00:34 2024-03-25T16:21:23.469 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:23.469 INFO:teuthology.orchestra.run.smithi193.stdout:Total 1.4 MB/s | 48 MB 00:35 2024-03-25T16:21:23.483 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T16:21:23.502 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T16:21:23.502 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T16:21:23.882 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T16:21:23.882 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T16:21:24.443 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T16:21:24.508 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:21:24.556 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:21:24.597 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:21:24.651 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:21:24.678 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:21:29.149 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:21:29.227 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:21:31.575 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:21:31.575 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:21:31.575 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:21:31.575 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:21:31.575 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:21:31.575 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:21:31.944 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:21:31.944 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:21:31.944 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T16:21:31.944 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:31.945 INFO:teuthology.orchestra.run.smithi193.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:21:31.945 INFO:teuthology.orchestra.run.smithi193.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:31.945 INFO:teuthology.orchestra.run.smithi193.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:21:31.945 INFO:teuthology.orchestra.run.smithi193.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:21:31.945 INFO:teuthology.orchestra.run.smithi193.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:21:31.945 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:21:31.945 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T16:21:32.256 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph 2024-03-25T16:21:32.830 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:34 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:21:32.992 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:21:32.994 INFO:teuthology.orchestra.run.smithi193.stdout:========================================================================================= 2024-03-25T16:21:32.995 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-25T16:21:32.995 INFO:teuthology.orchestra.run.smithi193.stdout:========================================================================================= 2024-03-25T16:21:32.995 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T16:21:32.995 INFO:teuthology.orchestra.run.smithi193.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:21:32.995 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-25T16:21:32.995 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:21:32.995 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:21:32.995 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:21:32.995 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:21:32.995 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:21:32.995 INFO:teuthology.orchestra.run.smithi193.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:21:32.995 INFO:teuthology.orchestra.run.smithi193.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:21:32.995 INFO:teuthology.orchestra.run.smithi193.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:21:32.996 INFO:teuthology.orchestra.run.smithi193.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:21:32.996 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:21:32.996 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:21:32.996 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:21:32.996 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:21:32.996 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:21:32.996 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:21:32.996 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:21:32.996 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:21:32.996 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:21:32.996 INFO:teuthology.orchestra.run.smithi193.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:21:32.996 INFO:teuthology.orchestra.run.smithi193.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:21:32.997 INFO:teuthology.orchestra.run.smithi193.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:21:32.997 INFO:teuthology.orchestra.run.smithi193.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:21:32.997 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:21:32.997 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:21:32.997 INFO:teuthology.orchestra.run.smithi193.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:21:32.997 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:21:32.997 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:21:32.997 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:21:32.997 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:21:32.997 INFO:teuthology.orchestra.run.smithi193.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:21:32.997 INFO:teuthology.orchestra.run.smithi193.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:21:32.997 INFO:teuthology.orchestra.run.smithi193.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:21:32.998 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:21:32.998 INFO:teuthology.orchestra.run.smithi193.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:21:32.998 INFO:teuthology.orchestra.run.smithi193.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:21:32.998 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:21:32.998 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T16:21:32.998 INFO:teuthology.orchestra.run.smithi193.stdout:========================================================================================= 2024-03-25T16:21:32.998 INFO:teuthology.orchestra.run.smithi193.stdout:Install 36 Packages 2024-03-25T16:21:32.998 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:21:32.999 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 30 M 2024-03-25T16:21:32.999 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 107 M 2024-03-25T16:21:32.999 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T16:21:33.489 INFO:teuthology.orchestra.run.smithi193.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 19 kB/s | 6.6 kB 00:00 2024-03-25T16:21:35.414 INFO:teuthology.orchestra.run.smithi193.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 959 kB/s | 2.1 MB 00:02 2024-03-25T16:21:36.864 INFO:teuthology.orchestra.run.smithi193.stdout:(3/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 1.4 MB/s | 4.7 MB 00:03 2024-03-25T16:21:37.048 INFO:teuthology.orchestra.run.smithi193.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.3 MB/s | 247 kB 00:00 2024-03-25T16:21:37.290 INFO:teuthology.orchestra.run.smithi193.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 772 kB/s | 187 kB 00:00 2024-03-25T16:21:37.332 INFO:teuthology.orchestra.run.smithi193.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T16:21:37.357 INFO:teuthology.orchestra.run.smithi193.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-25T16:21:37.382 INFO:teuthology.orchestra.run.smithi193.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.8 MB/s | 47 kB 00:00 2024-03-25T16:21:37.416 INFO:teuthology.orchestra.run.smithi193.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T16:21:37.458 INFO:teuthology.orchestra.run.smithi193.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-25T16:21:37.533 INFO:teuthology.orchestra.run.smithi193.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T16:21:37.575 INFO:teuthology.orchestra.run.smithi193.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-25T16:21:37.617 INFO:teuthology.orchestra.run.smithi193.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-25T16:21:37.668 INFO:teuthology.orchestra.run.smithi193.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-25T16:21:37.818 INFO:teuthology.orchestra.run.smithi193.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 833 kB/s | 125 kB 00:00 2024-03-25T16:21:38.043 INFO:teuthology.orchestra.run.smithi193.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 99 kB/s | 22 kB 00:00 2024-03-25T16:21:38.510 INFO:teuthology.orchestra.run.smithi193.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 324 kB/s | 151 kB 00:00 2024-03-25T16:21:38.627 INFO:teuthology.orchestra.run.smithi193.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 372 kB/s | 43 kB 00:00 2024-03-25T16:21:38.794 INFO:teuthology.orchestra.run.smithi193.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-03-25T16:21:39.002 INFO:teuthology.orchestra.run.smithi193.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-03-25T16:21:39.103 INFO:teuthology.orchestra.run.smithi193.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-03-25T16:21:39.195 INFO:teuthology.orchestra.run.smithi193.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 193 kB/s | 18 kB 00:00 2024-03-25T16:21:39.295 INFO:teuthology.orchestra.run.smithi193.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2024-03-25T16:21:39.362 INFO:teuthology.orchestra.run.smithi193.stdout:(24/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86 245 kB/s | 1.5 MB 00:06 2024-03-25T16:21:39.387 INFO:teuthology.orchestra.run.smithi193.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 212 kB/s | 19 kB 00:00 2024-03-25T16:21:39.488 INFO:teuthology.orchestra.run.smithi193.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 463 kB/s | 46 kB 00:00 2024-03-25T16:21:39.588 INFO:teuthology.orchestra.run.smithi193.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 788 kB/s | 79 kB 00:00 2024-03-25T16:21:39.605 INFO:teuthology.orchestra.run.smithi193.stdout:(28/36): python3-jaraco-text-3.2.0-6.el9.noarch 81 kB/s | 20 kB 00:00 2024-03-25T16:21:39.705 INFO:teuthology.orchestra.run.smithi193.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-03-25T16:21:39.722 INFO:teuthology.orchestra.run.smithi193.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 140 kB/s | 16 kB 00:00 2024-03-25T16:21:39.806 INFO:teuthology.orchestra.run.smithi193.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 893 kB/s | 90 kB 00:00 2024-03-25T16:21:39.882 INFO:teuthology.orchestra.run.smithi193.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 225 kB/s | 36 kB 00:00 2024-03-25T16:21:39.940 INFO:teuthology.orchestra.run.smithi193.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-03-25T16:21:40.049 INFO:teuthology.orchestra.run.smithi193.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 185 kB/s | 20 kB 00:00 2024-03-25T16:21:40.299 INFO:teuthology.orchestra.run.smithi193.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.0 MB/s | 427 kB 00:00 2024-03-25T16:21:45.515 INFO:teuthology.orchestra.run.smithi193.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 1.7 MB/s | 17 MB 00:10 2024-03-25T16:21:45.519 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:45.519 INFO:teuthology.orchestra.run.smithi193.stdout:Total 2.4 MB/s | 30 MB 00:12 2024-03-25T16:21:45.543 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T16:21:45.585 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T16:21:45.585 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T16:21:45.988 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T16:21:45.989 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T16:21:47.309 INFO:teuthology.orchestra.run.smithi066.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 849 kB/s | 47 MB 00:57 2024-03-25T16:21:47.311 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:47.312 INFO:teuthology.orchestra.run.smithi066.stdout:Total 861 kB/s | 48 MB 00:57 2024-03-25T16:21:47.324 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T16:21:47.328 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T16:21:47.346 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T16:21:47.346 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T16:21:47.384 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:21:47.426 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:21:47.462 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:21:47.523 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:21:47.569 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:21:47.602 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:21:47.693 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:21:47.742 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T16:21:47.742 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T16:21:47.763 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:21:47.813 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:21:47.884 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:21:47.939 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:21:47.977 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:21:48.026 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:21:48.081 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:21:48.265 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:21:48.315 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:21:48.340 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T16:21:48.354 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:21:48.388 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:21:48.418 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:21:48.419 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:21:48.465 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:21:48.494 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:21:48.506 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:21:48.529 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:21:48.557 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:21:48.562 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:21:48.579 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:21:48.602 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:21:48.638 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:21:48.672 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:21:48.761 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:21:48.803 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:21:48.875 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:21:49.011 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:21:49.137 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:21:49.303 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:21:49.334 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:21:49.360 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:21:49.360 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:21:49.360 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:21:49.360 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:21:49.360 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:21:49.360 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:21:50.801 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:21:50.827 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:21:50.828 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:21:50.828 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:21:50.828 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:21:50.828 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:21:50.828 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:21:51.269 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:21:51.292 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:21:51.292 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:21:51.292 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:21:51.292 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:21:51.292 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:21:51.292 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:21:51.503 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:21:51.526 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:21:51.526 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:21:51.526 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:21:51.526 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:21:51.526 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:21:51.526 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:21:51.591 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:21:53.296 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:21:53.345 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:21:53.346 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:21:53.346 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:21:53.346 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:21:53.346 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:21:53.346 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:21:53.346 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:21:53.346 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:21:53.346 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:21:53.346 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:21:53.346 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:21:53.346 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:21:53.346 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:21:53.346 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:21:53.347 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:21:53.347 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:21:53.347 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:21:53.347 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:21:53.347 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:21:53.347 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:21:53.347 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:21:53.347 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:21:53.347 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:21:53.347 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:21:53.347 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:21:53.347 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:21:53.347 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:21:53.347 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:21:53.348 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:21:53.348 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:21:53.348 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:21:53.348 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:21:53.348 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:21:53.348 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:21:53.348 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:21:53.348 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:21:53.382 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:21:54.455 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:21:54.455 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:21:54.455 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T16:21:54.455 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:54.455 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:54.456 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:54.456 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:21:54.456 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:54.456 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:54.456 INFO:teuthology.orchestra.run.smithi193.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:21:54.456 INFO:teuthology.orchestra.run.smithi193.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:21:54.457 INFO:teuthology.orchestra.run.smithi193.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:21:54.457 INFO:teuthology.orchestra.run.smithi193.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:21:54.457 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:21:54.457 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:21:54.457 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:21:54.457 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:21:54.458 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:21:54.458 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:21:54.458 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:21:54.458 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:21:54.458 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:21:54.458 INFO:teuthology.orchestra.run.smithi193.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:21:54.459 INFO:teuthology.orchestra.run.smithi193.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:21:54.459 INFO:teuthology.orchestra.run.smithi193.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:21:54.459 INFO:teuthology.orchestra.run.smithi193.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:21:54.459 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:21:54.459 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:21:54.460 INFO:teuthology.orchestra.run.smithi193.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:21:54.460 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:21:54.460 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:21:54.460 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:21:54.460 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:21:54.460 INFO:teuthology.orchestra.run.smithi193.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:21:54.461 INFO:teuthology.orchestra.run.smithi193.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:21:54.461 INFO:teuthology.orchestra.run.smithi193.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:21:54.461 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:21:54.461 INFO:teuthology.orchestra.run.smithi193.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:21:54.461 INFO:teuthology.orchestra.run.smithi193.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:21:54.461 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:21:54.461 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T16:21:54.819 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-base 2024-03-25T16:21:55.401 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:57 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:21:55.500 INFO:teuthology.orchestra.run.smithi193.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:21:55.552 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:21:55.553 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-03-25T16:21:55.553 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T16:21:55.611 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install cephadm 2024-03-25T16:21:55.994 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:21:55.994 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:21:55.994 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:21:55.994 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:21:55.995 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:21:55.995 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:21:56.168 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:58 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:21:56.317 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:21:56.318 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:21:56.318 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-25T16:21:56.318 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:21:56.318 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T16:21:56.318 INFO:teuthology.orchestra.run.smithi193.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:21:56.318 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:21:56.318 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T16:21:56.318 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:21:56.318 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T16:21:56.318 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:21:56.319 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 747 k 2024-03-25T16:21:56.319 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 754 k 2024-03-25T16:21:56.319 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T16:21:56.359 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:21:56.359 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:21:56.360 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T16:21:56.360 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:56.360 INFO:teuthology.orchestra.run.smithi066.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:21:56.360 INFO:teuthology.orchestra.run.smithi066.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:56.360 INFO:teuthology.orchestra.run.smithi066.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:21:56.360 INFO:teuthology.orchestra.run.smithi066.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:21:56.360 INFO:teuthology.orchestra.run.smithi066.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:21:56.360 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:21:56.360 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T16:21:56.654 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph 2024-03-25T16:21:57.243 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:01:59 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:21:57.412 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:21:57.414 INFO:teuthology.orchestra.run.smithi066.stdout:========================================================================================= 2024-03-25T16:21:57.414 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-03-25T16:21:57.414 INFO:teuthology.orchestra.run.smithi066.stdout:========================================================================================= 2024-03-25T16:21:57.414 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T16:21:57.414 INFO:teuthology.orchestra.run.smithi066.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:21:57.415 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-25T16:21:57.415 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:21:57.415 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:21:57.415 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:21:57.415 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:21:57.415 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:21:57.415 INFO:teuthology.orchestra.run.smithi066.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:21:57.415 INFO:teuthology.orchestra.run.smithi066.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:21:57.415 INFO:teuthology.orchestra.run.smithi066.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:21:57.415 INFO:teuthology.orchestra.run.smithi066.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:21:57.415 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:21:57.415 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:21:57.415 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:21:57.415 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:21:57.416 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:21:57.416 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:21:57.416 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:21:57.416 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:21:57.416 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:21:57.416 INFO:teuthology.orchestra.run.smithi066.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:21:57.417 INFO:teuthology.orchestra.run.smithi066.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:21:57.417 INFO:teuthology.orchestra.run.smithi066.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:21:57.417 INFO:teuthology.orchestra.run.smithi066.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:21:57.417 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:21:57.417 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:21:57.417 INFO:teuthology.orchestra.run.smithi066.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:21:57.417 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:21:57.417 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:21:57.417 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:21:57.417 INFO:teuthology.orchestra.run.smithi066.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:21:57.417 INFO:teuthology.orchestra.run.smithi066.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:21:57.418 INFO:teuthology.orchestra.run.smithi066.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:21:57.418 INFO:teuthology.orchestra.run.smithi066.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:21:57.418 INFO:teuthology.orchestra.run.smithi066.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:21:57.418 INFO:teuthology.orchestra.run.smithi066.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:21:57.418 INFO:teuthology.orchestra.run.smithi066.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:21:57.418 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:21:57.418 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T16:21:57.418 INFO:teuthology.orchestra.run.smithi066.stdout:========================================================================================= 2024-03-25T16:21:57.418 INFO:teuthology.orchestra.run.smithi066.stdout:Install 36 Packages 2024-03-25T16:21:57.418 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:21:57.419 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 30 M 2024-03-25T16:21:57.419 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 107 M 2024-03-25T16:21:57.419 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T16:21:58.028 INFO:teuthology.orchestra.run.smithi066.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 18 kB/s | 6.6 kB 00:00 2024-03-25T16:21:58.067 INFO:teuthology.orchestra.run.smithi193.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 428 kB/s | 747 kB 00:01 2024-03-25T16:21:58.068 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:58.068 INFO:teuthology.orchestra.run.smithi193.stdout:Total 427 kB/s | 747 kB 00:01 2024-03-25T16:21:58.068 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T16:21:58.074 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T16:21:58.074 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T16:21:58.080 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T16:21:58.081 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T16:21:58.152 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T16:21:58.603 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:21:58.689 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:21:59.084 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:21:59.402 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:21:59.402 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:21:59.402 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T16:21:59.402 INFO:teuthology.orchestra.run.smithi193.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:21:59.402 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:21:59.402 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T16:21:59.570 INFO:teuthology.orchestra.run.smithi066.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 797 kB/s | 1.5 MB 00:01 2024-03-25T16:21:59.617 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:22:00.213 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:02:02 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:22:00.370 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:22:00.371 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:22:00.372 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T16:22:00.372 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:22:00.372 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T16:22:00.372 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:22:00.372 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:00.373 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T16:22:00.373 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:22:00.373 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T16:22:00.373 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:00.373 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 145 k 2024-03-25T16:22:00.373 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 436 k 2024-03-25T16:22:00.374 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T16:22:00.812 INFO:teuthology.orchestra.run.smithi066.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 689 kB/s | 2.1 MB 00:03 2024-03-25T16:22:01.112 INFO:teuthology.orchestra.run.smithi066.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 825 kB/s | 247 kB 00:00 2024-03-25T16:22:01.158 INFO:teuthology.orchestra.run.smithi193.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 185 kB/s | 145 kB 00:00 2024-03-25T16:22:01.158 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:01.159 INFO:teuthology.orchestra.run.smithi193.stdout:Total 185 kB/s | 145 kB 00:00 2024-03-25T16:22:01.159 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T16:22:01.167 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T16:22:01.167 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T16:22:01.219 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T16:22:01.220 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T16:22:01.237 INFO:teuthology.orchestra.run.smithi066.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-25T16:22:01.288 INFO:teuthology.orchestra.run.smithi066.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-25T16:22:01.329 INFO:teuthology.orchestra.run.smithi066.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 759 kB/s | 32 kB 00:00 2024-03-25T16:22:01.358 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T16:22:01.363 INFO:teuthology.orchestra.run.smithi066.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T16:22:01.405 INFO:teuthology.orchestra.run.smithi066.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 929 kB/s | 39 kB 00:00 2024-03-25T16:22:01.417 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:22:01.442 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:22:01.442 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:22:01.442 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:22:01.442 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:01.446 INFO:teuthology.orchestra.run.smithi066.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-25T16:22:01.530 INFO:teuthology.orchestra.run.smithi066.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-25T16:22:01.574 INFO:teuthology.orchestra.run.smithi066.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-25T16:22:01.614 INFO:teuthology.orchestra.run.smithi066.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-25T16:22:01.656 INFO:teuthology.orchestra.run.smithi066.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T16:22:01.884 INFO:teuthology.orchestra.run.smithi066.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 548 kB/s | 125 kB 00:00 2024-03-25T16:22:01.932 INFO:teuthology.orchestra.run.smithi066.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 475 kB/s | 22 kB 00:00 2024-03-25T16:22:02.299 INFO:teuthology.orchestra.run.smithi066.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 412 kB/s | 151 kB 00:00 2024-03-25T16:22:02.391 INFO:teuthology.orchestra.run.smithi066.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 474 kB/s | 43 kB 00:00 2024-03-25T16:22:02.399 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:22:02.400 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:02.400 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T16:22:02.400 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:02.400 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:02.400 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T16:22:02.516 INFO:teuthology.orchestra.run.smithi066.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-03-25T16:22:02.608 INFO:teuthology.orchestra.run.smithi066.stdout:(20/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 1.0 MB/s | 4.7 MB 00:04 2024-03-25T16:22:02.641 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mgr 2024-03-25T16:22:02.692 INFO:teuthology.orchestra.run.smithi066.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.0 MB/s | 358 kB 00:00 2024-03-25T16:22:02.734 INFO:teuthology.orchestra.run.smithi066.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 85 kB/s | 11 kB 00:00 2024-03-25T16:22:02.775 INFO:teuthology.orchestra.run.smithi066.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 211 kB/s | 18 kB 00:00 2024-03-25T16:22:02.843 INFO:teuthology.orchestra.run.smithi066.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 213 kB/s | 23 kB 00:00 2024-03-25T16:22:02.868 INFO:teuthology.orchestra.run.smithi066.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 211 kB/s | 19 kB 00:00 2024-03-25T16:22:02.943 INFO:teuthology.orchestra.run.smithi066.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2024-03-25T16:22:02.968 INFO:teuthology.orchestra.run.smithi066.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 462 kB/s | 46 kB 00:00 2024-03-25T16:22:03.068 INFO:teuthology.orchestra.run.smithi066.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-03-25T16:22:03.119 INFO:teuthology.orchestra.run.smithi066.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 449 kB/s | 79 kB 00:00 2024-03-25T16:22:03.152 INFO:teuthology.orchestra.run.smithi066.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 196 kB/s | 16 kB 00:00 2024-03-25T16:22:03.211 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:02:05 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:22:03.236 INFO:teuthology.orchestra.run.smithi066.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 429 kB/s | 36 kB 00:00 2024-03-25T16:22:03.261 INFO:teuthology.orchestra.run.smithi066.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 633 kB/s | 90 kB 00:00 2024-03-25T16:22:03.307 INFO:teuthology.orchestra.run.smithi193.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:22:03.328 INFO:teuthology.orchestra.run.smithi066.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-25T16:22:03.359 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:22:03.360 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-03-25T16:22:03.360 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T16:22:03.412 INFO:teuthology.orchestra.run.smithi066.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 240 kB/s | 20 kB 00:00 2024-03-25T16:22:03.417 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:22:03.462 INFO:teuthology.orchestra.run.smithi066.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.1 MB/s | 427 kB 00:00 2024-03-25T16:22:03.975 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:02:05 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:22:04.124 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:22:04.125 INFO:teuthology.orchestra.run.smithi193.stdout:====================================================================================== 2024-03-25T16:22:04.125 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-25T16:22:04.125 INFO:teuthology.orchestra.run.smithi193.stdout:====================================================================================== 2024-03-25T16:22:04.125 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T16:22:04.126 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:22:04.126 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-25T16:22:04.126 INFO:teuthology.orchestra.run.smithi193.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:22:04.126 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:22:04.126 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:22:04.126 INFO:teuthology.orchestra.run.smithi193.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:22:04.126 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:22:04.126 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:22:04.126 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:22:04.126 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:22:04.126 INFO:teuthology.orchestra.run.smithi193.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:22:04.126 INFO:teuthology.orchestra.run.smithi193.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:22:04.126 INFO:teuthology.orchestra.run.smithi193.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:22:04.126 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:04.127 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T16:22:04.127 INFO:teuthology.orchestra.run.smithi193.stdout:====================================================================================== 2024-03-25T16:22:04.127 INFO:teuthology.orchestra.run.smithi193.stdout:Install 12 Packages 2024-03-25T16:22:04.127 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:04.127 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 6.9 M 2024-03-25T16:22:04.127 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 26 M 2024-03-25T16:22:04.127 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T16:22:04.972 INFO:teuthology.orchestra.run.smithi193.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 30 kB/s | 15 kB 00:00 2024-03-25T16:22:05.072 INFO:teuthology.orchestra.run.smithi193.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 51 kB/s | 31 kB 00:00 2024-03-25T16:22:05.139 INFO:teuthology.orchestra.run.smithi193.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.0 MB/s | 1.0 MB 00:00 2024-03-25T16:22:05.164 INFO:teuthology.orchestra.run.smithi193.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.5 MB/s | 238 kB 00:00 2024-03-25T16:22:05.381 INFO:teuthology.orchestra.run.smithi193.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.5 MB/s | 863 kB 00:00 2024-03-25T16:22:05.523 INFO:teuthology.orchestra.run.smithi193.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.5 MB/s | 548 kB 00:00 2024-03-25T16:22:05.549 INFO:teuthology.orchestra.run.smithi193.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 116 kB/s | 19 kB 00:00 2024-03-25T16:22:05.699 INFO:teuthology.orchestra.run.smithi193.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 963 kB/s | 144 kB 00:00 2024-03-25T16:22:05.782 INFO:teuthology.orchestra.run.smithi193.stdout:(9/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 7.9 MB/s | 2.0 MB 00:00 2024-03-25T16:22:05.808 INFO:teuthology.orchestra.run.smithi193.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 282 kB/s | 31 kB 00:00 2024-03-25T16:22:05.900 INFO:teuthology.orchestra.run.smithi193.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2024-03-25T16:22:09.949 INFO:teuthology.orchestra.run.smithi193.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 332 kB/s | 1.8 MB 00:05 2024-03-25T16:22:09.953 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:09.953 INFO:teuthology.orchestra.run.smithi193.stdout:Total 1.2 MB/s | 6.9 MB 00:05 2024-03-25T16:22:09.973 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T16:22:09.992 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T16:22:09.993 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T16:22:10.164 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T16:22:10.165 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T16:22:10.681 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T16:22:10.766 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:22:10.802 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:22:10.848 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:22:10.879 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:22:10.982 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:22:11.112 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:22:11.207 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:22:11.251 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:22:11.279 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:22:11.311 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:22:11.553 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:22:11.577 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:22:12.315 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:22:12.315 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:22:12.315 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:22:12.315 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:22:12.315 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:22:12.316 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:22:12.316 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:22:12.316 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:22:12.316 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:22:12.316 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:22:12.316 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:22:12.316 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:22:12.653 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:22:12.653 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:12.653 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T16:22:12.653 INFO:teuthology.orchestra.run.smithi193.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:22:12.654 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:12.654 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:12.654 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:12.654 INFO:teuthology.orchestra.run.smithi193.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:22:12.654 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:22:12.654 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:22:12.654 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:22:12.654 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:22:12.654 INFO:teuthology.orchestra.run.smithi193.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:22:12.654 INFO:teuthology.orchestra.run.smithi193.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:22:12.654 INFO:teuthology.orchestra.run.smithi193.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:22:12.654 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:12.654 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T16:22:12.899 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:22:13.466 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:02:15 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:22:13.617 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:22:13.618 INFO:teuthology.orchestra.run.smithi193.stdout:============================================================================================ 2024-03-25T16:22:13.618 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-25T16:22:13.618 INFO:teuthology.orchestra.run.smithi193.stdout:============================================================================================ 2024-03-25T16:22:13.618 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T16:22:13.618 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:22:13.618 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-25T16:22:13.618 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:22:13.619 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:22:13.619 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:22:13.619 INFO:teuthology.orchestra.run.smithi193.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:22:13.619 INFO:teuthology.orchestra.run.smithi193.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:22:13.619 INFO:teuthology.orchestra.run.smithi193.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:22:13.619 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:22:13.619 INFO:teuthology.orchestra.run.smithi193.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:22:13.619 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:22:13.619 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:22:13.619 INFO:teuthology.orchestra.run.smithi193.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:22:13.619 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:13.619 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T16:22:13.619 INFO:teuthology.orchestra.run.smithi193.stdout:============================================================================================ 2024-03-25T16:22:13.619 INFO:teuthology.orchestra.run.smithi193.stdout:Install 12 Packages 2024-03-25T16:22:13.620 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:13.620 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 37 M 2024-03-25T16:22:13.620 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 209 M 2024-03-25T16:22:13.620 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T16:22:13.999 INFO:teuthology.orchestra.run.smithi193.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 183 kB/s | 32 kB 00:00 2024-03-25T16:22:14.049 INFO:teuthology.orchestra.run.smithi193.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 342 kB/s | 17 kB 00:00 2024-03-25T16:22:14.108 INFO:teuthology.orchestra.run.smithi193.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 586 kB/s | 34 kB 00:00 2024-03-25T16:22:14.533 INFO:teuthology.orchestra.run.smithi193.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 11 MB/s | 4.7 MB 00:00 2024-03-25T16:22:14.691 INFO:teuthology.orchestra.run.smithi193.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 30 MB/s | 4.8 MB 00:00 2024-03-25T16:22:14.734 INFO:teuthology.orchestra.run.smithi193.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.2 MB/s | 217 kB 00:00 2024-03-25T16:22:14.800 INFO:teuthology.orchestra.run.smithi193.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.1 MB/s | 3.0 MB 00:00 2024-03-25T16:22:15.126 INFO:teuthology.orchestra.run.smithi193.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2024-03-25T16:22:15.292 INFO:teuthology.orchestra.run.smithi193.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 803 kB 00:00 2024-03-25T16:22:15.335 INFO:teuthology.orchestra.run.smithi193.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 942 kB/s | 195 kB 00:00 2024-03-25T16:22:15.551 INFO:teuthology.orchestra.run.smithi193.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 949 kB/s | 245 kB 00:00 2024-03-25T16:22:17.769 INFO:teuthology.orchestra.run.smithi066.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 944 kB/s | 17 MB 00:18 2024-03-25T16:22:17.773 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:17.773 INFO:teuthology.orchestra.run.smithi066.stdout:Total 1.5 MB/s | 30 MB 00:20 2024-03-25T16:22:17.798 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T16:22:17.846 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T16:22:17.846 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T16:22:18.260 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T16:22:18.261 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T16:22:19.602 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T16:22:19.663 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:22:19.705 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:22:19.732 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:22:19.796 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:22:19.839 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:22:19.872 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:22:19.976 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:22:20.052 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:22:20.101 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:22:20.173 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:22:20.227 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:22:20.264 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:22:20.314 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:22:20.369 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:22:20.560 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:22:20.601 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:22:20.640 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:22:20.683 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:22:20.713 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:22:20.791 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:22:20.838 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:22:20.873 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:22:20.914 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:22:20.949 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:22:20.991 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:22:21.083 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:22:21.122 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:22:21.197 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:22:21.336 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:22:21.460 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:22:21.630 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:22:21.660 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:22:21.687 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:22:21.687 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:22:21.687 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:22:21.687 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:22:21.687 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:22:21.687 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:22:23.202 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:22:23.230 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:22:23.230 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:22:23.230 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:22:23.230 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:22:23.230 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:22:23.231 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:22:23.259 INFO:teuthology.orchestra.run.smithi193.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 802 kB/s | 7.4 MB 00:09 2024-03-25T16:22:23.264 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:23.264 INFO:teuthology.orchestra.run.smithi193.stdout:Total 3.8 MB/s | 37 MB 00:09 2024-03-25T16:22:23.308 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T16:22:23.332 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T16:22:23.333 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T16:22:23.687 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:22:23.692 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T16:22:23.692 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T16:22:23.711 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:22:23.711 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:22:23.711 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:22:23.711 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:22:23.712 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:22:23.712 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:22:23.920 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:22:23.945 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:22:23.945 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:22:23.945 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:22:23.945 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:22:23.945 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:22:23.945 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:22:24.021 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:22:24.403 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T16:22:24.465 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:22:24.496 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:22:24.592 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:22:24.624 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:22:24.801 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:22:24.832 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:22:24.909 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:22:25.568 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:22:25.678 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:22:25.791 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:22:25.791 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:22:25.791 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:22:25.791 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:22:25.792 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:22:25.792 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:22:25.792 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:22:25.792 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:22:25.792 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:22:25.792 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:22:25.792 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:22:25.792 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:22:25.792 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:22:25.792 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:22:25.792 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:22:25.792 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:22:25.792 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:22:25.793 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:22:25.793 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:22:25.793 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:22:25.793 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:22:25.793 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:22:25.793 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:22:25.793 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:22:25.793 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:22:25.793 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:22:25.793 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:22:25.793 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:22:25.794 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:22:25.795 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:22:25.795 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:22:25.795 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:22:25.795 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:22:25.795 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:22:25.795 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:22:25.795 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:22:26.851 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:22:26.851 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:22:26.851 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T16:22:26.851 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:26.852 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:26.852 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:26.852 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:26.852 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:26.852 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:26.852 INFO:teuthology.orchestra.run.smithi066.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:22:26.852 INFO:teuthology.orchestra.run.smithi066.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:22:26.852 INFO:teuthology.orchestra.run.smithi066.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:22:26.852 INFO:teuthology.orchestra.run.smithi066.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:22:26.852 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:22:26.852 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:22:26.852 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:22:26.852 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:22:26.852 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:22:26.853 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:22:26.853 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:22:26.853 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:22:26.853 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:22:26.853 INFO:teuthology.orchestra.run.smithi066.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:22:26.853 INFO:teuthology.orchestra.run.smithi066.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:22:26.853 INFO:teuthology.orchestra.run.smithi066.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:22:26.853 INFO:teuthology.orchestra.run.smithi066.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:22:26.853 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:22:26.853 INFO:teuthology.orchestra.run.smithi066.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:22:26.853 INFO:teuthology.orchestra.run.smithi066.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:22:26.853 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:22:26.853 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:22:26.853 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:22:26.854 INFO:teuthology.orchestra.run.smithi066.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:22:26.854 INFO:teuthology.orchestra.run.smithi066.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:22:26.854 INFO:teuthology.orchestra.run.smithi066.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:22:26.854 INFO:teuthology.orchestra.run.smithi066.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:22:26.854 INFO:teuthology.orchestra.run.smithi066.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:22:26.854 INFO:teuthology.orchestra.run.smithi066.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:22:26.854 INFO:teuthology.orchestra.run.smithi066.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:22:26.854 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:22:26.854 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T16:22:26.915 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:22:27.231 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-base 2024-03-25T16:22:27.835 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:02:29 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:22:27.936 INFO:teuthology.orchestra.run.smithi066.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:22:27.950 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:22:27.974 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:22:27.991 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:22:27.992 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-03-25T16:22:27.992 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T16:22:28.066 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install cephadm 2024-03-25T16:22:28.653 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:02:30 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:22:28.809 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:22:28.810 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:22:28.810 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-03-25T16:22:28.810 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:22:28.810 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T16:22:28.810 INFO:teuthology.orchestra.run.smithi066.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:22:28.810 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:22:28.810 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T16:22:28.810 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:22:28.810 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-25T16:22:28.810 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:22:28.810 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 747 k 2024-03-25T16:22:28.810 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 754 k 2024-03-25T16:22:28.811 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T16:22:30.044 INFO:teuthology.orchestra.run.smithi066.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 606 kB/s | 747 kB 00:01 2024-03-25T16:22:30.045 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:30.045 INFO:teuthology.orchestra.run.smithi066.stdout:Total 605 kB/s | 747 kB 00:01 2024-03-25T16:22:30.046 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T16:22:30.052 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T16:22:30.052 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T16:22:30.059 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T16:22:30.059 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T16:22:30.136 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T16:22:30.357 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:22:30.357 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:22:30.358 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:22:30.358 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:22:30.358 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:22:30.358 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:22:30.358 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:22:30.358 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:22:30.358 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:22:30.358 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:22:30.358 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:22:30.358 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:22:30.541 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:22:30.711 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:22:30.781 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:22:30.781 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:30.781 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T16:22:30.781 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:30.781 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:22:30.781 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:22:30.781 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:22:30.781 INFO:teuthology.orchestra.run.smithi193.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:22:30.781 INFO:teuthology.orchestra.run.smithi193.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:22:30.781 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:22:30.781 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:22:30.782 INFO:teuthology.orchestra.run.smithi193.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:22:30.782 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:22:30.782 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:22:30.782 INFO:teuthology.orchestra.run.smithi193.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:22:30.782 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:30.782 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T16:22:31.161 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:22:31.178 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:22:31.448 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:22:31.448 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:22:31.448 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T16:22:31.448 INFO:teuthology.orchestra.run.smithi066.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:31.448 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:22:31.449 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T16:22:31.651 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:22:31.760 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:02:33 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:22:31.922 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:22:31.923 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================== 2024-03-25T16:22:31.923 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-25T16:22:31.924 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================== 2024-03-25T16:22:31.924 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T16:22:31.924 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:22:31.924 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-25T16:22:31.924 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:22:31.924 INFO:teuthology.orchestra.run.smithi193.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:22:31.924 INFO:teuthology.orchestra.run.smithi193.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:22:31.924 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:22:31.924 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:22:31.924 INFO:teuthology.orchestra.run.smithi193.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:22:31.924 INFO:teuthology.orchestra.run.smithi193.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:22:31.924 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:22:31.924 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:22:31.925 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:22:31.925 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:22:31.925 INFO:teuthology.orchestra.run.smithi193.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:22:31.925 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:31.925 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T16:22:31.925 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================== 2024-03-25T16:22:31.925 INFO:teuthology.orchestra.run.smithi193.stdout:Install 13 Packages 2024-03-25T16:22:31.925 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:31.926 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 2.0 M 2024-03-25T16:22:31.926 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 26 M 2024-03-25T16:22:31.926 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T16:22:32.249 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:02:34 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:22:32.404 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:22:32.405 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:22:32.405 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-25T16:22:32.405 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:22:32.405 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T16:22:32.405 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:22:32.405 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:22:32.406 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T16:22:32.406 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:22:32.406 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-25T16:22:32.406 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:22:32.406 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 145 k 2024-03-25T16:22:32.406 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 436 k 2024-03-25T16:22:32.406 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T16:22:32.434 INFO:teuthology.orchestra.run.smithi193.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 195 kB/s | 24 kB 00:00 2024-03-25T16:22:32.459 INFO:teuthology.orchestra.run.smithi193.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 120 kB/s | 18 kB 00:00 2024-03-25T16:22:32.509 INFO:teuthology.orchestra.run.smithi193.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-25T16:22:32.534 INFO:teuthology.orchestra.run.smithi193.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T16:22:32.560 INFO:teuthology.orchestra.run.smithi193.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T16:22:32.585 INFO:teuthology.orchestra.run.smithi193.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2024-03-25T16:22:32.769 INFO:teuthology.orchestra.run.smithi193.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 113 kB/s | 52 kB 00:00 2024-03-25T16:22:32.867 INFO:teuthology.orchestra.run.smithi193.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 137 kB/s | 32 kB 00:00 2024-03-25T16:22:32.868 INFO:teuthology.orchestra.run.smithi193.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 66 kB/s | 15 kB 00:00 2024-03-25T16:22:32.919 INFO:teuthology.orchestra.run.smithi193.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.3 MB/s | 203 kB 00:00 2024-03-25T16:22:32.995 INFO:teuthology.orchestra.run.smithi193.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 322 kB/s | 59 kB 00:00 2024-03-25T16:22:33.091 INFO:teuthology.orchestra.run.smithi066.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 212 kB/s | 145 kB 00:00 2024-03-25T16:22:33.091 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:33.092 INFO:teuthology.orchestra.run.smithi066.stdout:Total 212 kB/s | 145 kB 00:00 2024-03-25T16:22:33.092 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T16:22:33.101 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T16:22:33.102 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T16:22:33.129 INFO:teuthology.orchestra.run.smithi193.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 430 kB/s | 90 kB 00:00 2024-03-25T16:22:33.159 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T16:22:33.160 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T16:22:33.162 INFO:teuthology.orchestra.run.smithi193.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.8 MB/s | 1.0 MB 00:00 2024-03-25T16:22:33.165 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:33.166 INFO:teuthology.orchestra.run.smithi193.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2024-03-25T16:22:33.166 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T16:22:33.179 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T16:22:33.179 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T16:22:33.268 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T16:22:33.269 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T16:22:33.318 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T16:22:33.400 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:22:33.427 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:22:33.427 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:22:33.427 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:22:33.427 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:22:33.789 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T16:22:33.958 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:22:34.005 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:22:34.061 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:22:34.106 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:22:34.156 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:22:34.250 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:22:34.350 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:22:34.399 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:22:34.434 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:22:34.435 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:22:34.435 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T16:22:34.435 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:34.435 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:22:34.435 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T16:22:34.667 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-mgr 2024-03-25T16:22:35.115 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:22:35.163 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:22:35.197 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:22:35.245 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:22:35.259 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:02:37 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:22:35.267 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:22:35.360 INFO:teuthology.orchestra.run.smithi066.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:22:35.416 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:22:35.417 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-03-25T16:22:35.417 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T16:22:35.489 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:22:36.047 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:22:36.048 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:22:36.048 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:22:36.048 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:22:36.048 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:22:36.048 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:22:36.048 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:22:36.048 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:22:36.048 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:22:36.048 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:22:36.048 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:22:36.048 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:22:36.048 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:22:36.077 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:02:38 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:22:36.232 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:22:36.233 INFO:teuthology.orchestra.run.smithi066.stdout:====================================================================================== 2024-03-25T16:22:36.233 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-03-25T16:22:36.233 INFO:teuthology.orchestra.run.smithi066.stdout:====================================================================================== 2024-03-25T16:22:36.233 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T16:22:36.234 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:22:36.234 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-25T16:22:36.234 INFO:teuthology.orchestra.run.smithi066.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:22:36.234 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:22:36.234 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:22:36.234 INFO:teuthology.orchestra.run.smithi066.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:22:36.234 INFO:teuthology.orchestra.run.smithi066.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:22:36.234 INFO:teuthology.orchestra.run.smithi066.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:22:36.234 INFO:teuthology.orchestra.run.smithi066.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:22:36.234 INFO:teuthology.orchestra.run.smithi066.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:22:36.234 INFO:teuthology.orchestra.run.smithi066.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:22:36.234 INFO:teuthology.orchestra.run.smithi066.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:22:36.235 INFO:teuthology.orchestra.run.smithi066.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:22:36.235 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:22:36.235 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T16:22:36.235 INFO:teuthology.orchestra.run.smithi066.stdout:====================================================================================== 2024-03-25T16:22:36.235 INFO:teuthology.orchestra.run.smithi066.stdout:Install 12 Packages 2024-03-25T16:22:36.235 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:22:36.235 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 6.9 M 2024-03-25T16:22:36.236 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 26 M 2024-03-25T16:22:36.236 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T16:22:36.392 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:22:36.392 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:36.392 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T16:22:36.392 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:36.392 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:22:36.392 INFO:teuthology.orchestra.run.smithi193.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:22:36.393 INFO:teuthology.orchestra.run.smithi193.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:22:36.393 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:22:36.393 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:22:36.393 INFO:teuthology.orchestra.run.smithi193.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:22:36.393 INFO:teuthology.orchestra.run.smithi193.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:22:36.393 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:22:36.393 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:22:36.393 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:22:36.393 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:22:36.393 INFO:teuthology.orchestra.run.smithi193.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:22:36.393 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:36.393 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T16:22:36.672 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:22:37.239 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:02:39 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:22:37.294 INFO:teuthology.orchestra.run.smithi066.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 36 kB/s | 15 kB 00:00 2024-03-25T16:22:37.390 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:22:37.395 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================== 2024-03-25T16:22:37.395 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-25T16:22:37.395 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================== 2024-03-25T16:22:37.395 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T16:22:37.395 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:22:37.395 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-25T16:22:37.395 INFO:teuthology.orchestra.run.smithi193.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:22:37.395 INFO:teuthology.orchestra.run.smithi193.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:22:37.395 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:22:37.395 INFO:teuthology.orchestra.run.smithi193.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:22:37.396 INFO:teuthology.orchestra.run.smithi193.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:22:37.396 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:37.396 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T16:22:37.396 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================== 2024-03-25T16:22:37.396 INFO:teuthology.orchestra.run.smithi193.stdout:Install 6 Packages 2024-03-25T16:22:37.396 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:37.396 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 6.8 M 2024-03-25T16:22:37.396 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 33 M 2024-03-25T16:22:37.396 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T16:22:37.411 INFO:teuthology.orchestra.run.smithi066.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 58 kB/s | 31 kB 00:00 2024-03-25T16:22:37.436 INFO:teuthology.orchestra.run.smithi066.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 7.1 MB/s | 1.0 MB 00:00 2024-03-25T16:22:37.461 INFO:teuthology.orchestra.run.smithi066.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 4.7 MB/s | 238 kB 00:00 2024-03-25T16:22:37.874 INFO:teuthology.orchestra.run.smithi193.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-03-25T16:22:37.913 INFO:teuthology.orchestra.run.smithi066.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.2 MB/s | 548 kB 00:00 2024-03-25T16:22:37.977 INFO:teuthology.orchestra.run.smithi193.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-25T16:22:38.008 INFO:teuthology.orchestra.run.smithi066.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 212 kB/s | 19 kB 00:00 2024-03-25T16:22:38.124 INFO:teuthology.orchestra.run.smithi193.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 385 kB/s | 58 kB 00:00 2024-03-25T16:22:38.146 INFO:teuthology.orchestra.run.smithi066.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.2 MB/s | 863 kB 00:00 2024-03-25T16:22:38.192 INFO:teuthology.orchestra.run.smithi193.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 763 kB/s | 51 kB 00:00 2024-03-25T16:22:38.230 INFO:teuthology.orchestra.run.smithi066.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 9.1 MB/s | 2.0 MB 00:00 2024-03-25T16:22:38.241 INFO:teuthology.orchestra.run.smithi193.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.5 MB/s | 548 kB 00:00 2024-03-25T16:22:38.255 INFO:teuthology.orchestra.run.smithi066.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.3 MB/s | 144 kB 00:00 2024-03-25T16:22:38.288 INFO:teuthology.orchestra.run.smithi066.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 529 kB/s | 31 kB 00:00 2024-03-25T16:22:38.313 INFO:teuthology.orchestra.run.smithi066.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.2 MB/s | 188 kB 00:00 2024-03-25T16:22:38.391 INFO:teuthology.orchestra.run.smithi193.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 196 kB/s | 138 kB 00:00 2024-03-25T16:22:38.393 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:38.393 INFO:teuthology.orchestra.run.smithi193.stdout:Total 6.8 MB/s | 6.8 MB 00:01 2024-03-25T16:22:38.406 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T16:22:38.417 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T16:22:38.417 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T16:22:38.504 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T16:22:38.505 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T16:22:38.804 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T16:22:38.974 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:22:39.034 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:22:39.546 INFO:teuthology.orchestra.run.smithi066.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 681 kB/s | 1.8 MB 00:02 2024-03-25T16:22:39.549 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:22:39.551 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:39.551 INFO:teuthology.orchestra.run.smithi066.stdout:Total 2.1 MB/s | 6.9 MB 00:03 2024-03-25T16:22:39.571 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T16:22:39.592 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T16:22:39.592 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T16:22:39.636 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:22:39.707 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:22:39.731 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:22:39.779 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T16:22:39.780 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T16:22:40.317 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T16:22:40.398 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:22:40.433 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:22:40.473 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:22:40.501 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:22:40.505 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:22:40.505 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:22:40.505 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:22:40.505 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:22:40.505 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:22:40.505 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:22:40.606 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:22:40.745 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:22:40.837 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:22:40.874 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:22:40.874 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:40.874 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T16:22:40.874 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:40.874 INFO:teuthology.orchestra.run.smithi193.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:22:40.874 INFO:teuthology.orchestra.run.smithi193.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:22:40.874 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:22:40.874 INFO:teuthology.orchestra.run.smithi193.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:22:40.874 INFO:teuthology.orchestra.run.smithi193.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:22:40.874 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:40.875 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T16:22:40.882 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:22:40.909 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:22:40.941 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:22:41.139 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-fuse 2024-03-25T16:22:41.201 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:22:41.232 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:22:41.709 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:02:43 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:22:41.858 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:22:41.859 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:22:41.859 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-25T16:22:41.859 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:22:41.859 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T16:22:41.859 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:22:41.859 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-25T16:22:41.859 INFO:teuthology.orchestra.run.smithi193.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:22:41.859 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:41.859 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T16:22:41.860 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:22:41.860 INFO:teuthology.orchestra.run.smithi193.stdout:Install 2 Packages 2024-03-25T16:22:41.860 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:41.860 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 964 k 2024-03-25T16:22:41.860 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 2.8 M 2024-03-25T16:22:41.860 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T16:22:41.985 INFO:teuthology.orchestra.run.smithi193.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 628 kB/s | 79 kB 00:00 2024-03-25T16:22:42.142 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:22:42.142 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:22:42.142 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:22:42.142 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:22:42.143 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:22:42.143 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:22:42.143 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:22:42.143 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:22:42.143 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:22:42.143 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:22:42.143 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:22:42.143 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:22:42.498 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:22:42.498 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:22:42.498 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T16:22:42.498 INFO:teuthology.orchestra.run.smithi066.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:22:42.498 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:42.498 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:42.498 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:42.498 INFO:teuthology.orchestra.run.smithi066.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:22:42.498 INFO:teuthology.orchestra.run.smithi066.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:22:42.498 INFO:teuthology.orchestra.run.smithi066.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:22:42.498 INFO:teuthology.orchestra.run.smithi066.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:22:42.499 INFO:teuthology.orchestra.run.smithi066.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:22:42.499 INFO:teuthology.orchestra.run.smithi066.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:22:42.499 INFO:teuthology.orchestra.run.smithi066.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:22:42.499 INFO:teuthology.orchestra.run.smithi066.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:22:42.499 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:22:42.499 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T16:22:42.768 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:22:43.357 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:02:45 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:22:43.514 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:22:43.515 INFO:teuthology.orchestra.run.smithi066.stdout:============================================================================================ 2024-03-25T16:22:43.515 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-03-25T16:22:43.515 INFO:teuthology.orchestra.run.smithi066.stdout:============================================================================================ 2024-03-25T16:22:43.515 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T16:22:43.515 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:22:43.515 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-25T16:22:43.515 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:22:43.515 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:22:43.515 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:22:43.515 INFO:teuthology.orchestra.run.smithi066.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:22:43.516 INFO:teuthology.orchestra.run.smithi066.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:22:43.516 INFO:teuthology.orchestra.run.smithi066.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:22:43.516 INFO:teuthology.orchestra.run.smithi066.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:22:43.516 INFO:teuthology.orchestra.run.smithi066.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:22:43.516 INFO:teuthology.orchestra.run.smithi066.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:22:43.516 INFO:teuthology.orchestra.run.smithi066.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:22:43.516 INFO:teuthology.orchestra.run.smithi066.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:22:43.516 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:22:43.516 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T16:22:43.516 INFO:teuthology.orchestra.run.smithi066.stdout:============================================================================================ 2024-03-25T16:22:43.516 INFO:teuthology.orchestra.run.smithi066.stdout:Install 12 Packages 2024-03-25T16:22:43.516 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:22:43.517 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 37 M 2024-03-25T16:22:43.517 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 209 M 2024-03-25T16:22:43.517 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T16:22:43.720 INFO:teuthology.orchestra.run.smithi066.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 270 kB/s | 32 kB 00:00 2024-03-25T16:22:43.778 INFO:teuthology.orchestra.run.smithi066.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 530 kB/s | 17 kB 00:00 2024-03-25T16:22:43.812 INFO:teuthology.orchestra.run.smithi066.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T16:22:43.919 INFO:teuthology.orchestra.run.smithi193.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 430 kB/s | 885 kB 00:02 2024-03-25T16:22:43.921 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:43.921 INFO:teuthology.orchestra.run.smithi193.stdout:Total 467 kB/s | 964 kB 00:02 2024-03-25T16:22:43.921 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T16:22:43.946 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T16:22:43.946 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T16:22:44.004 INFO:teuthology.orchestra.run.smithi066.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2024-03-25T16:22:44.020 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T16:22:44.020 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T16:22:44.170 INFO:teuthology.orchestra.run.smithi066.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 4.8 MB 00:00 2024-03-25T16:22:44.200 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T16:22:44.212 INFO:teuthology.orchestra.run.smithi066.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.2 MB/s | 217 kB 00:00 2024-03-25T16:22:44.329 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:22:44.379 INFO:teuthology.orchestra.run.smithi066.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.8 MB/s | 3.0 MB 00:00 2024-03-25T16:22:44.403 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:22:44.596 INFO:teuthology.orchestra.run.smithi066.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-03-25T16:22:45.113 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:22:45.113 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:22:45.428 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:22:45.428 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:45.428 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T16:22:45.429 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:22:45.429 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:45.429 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T16:22:45.430 INFO:teuthology.orchestra.run.smithi066.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 234 kB/s | 195 kB 00:00 2024-03-25T16:22:45.691 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-volume 2024-03-25T16:22:46.130 INFO:teuthology.orchestra.run.smithi066.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 459 kB/s | 803 kB 00:01 2024-03-25T16:22:46.268 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:02:48 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:22:46.348 INFO:teuthology.orchestra.run.smithi066.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 268 kB/s | 245 kB 00:00 2024-03-25T16:22:46.423 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:22:46.424 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:22:46.424 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-25T16:22:46.424 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:22:46.424 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T16:22:46.424 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:22:46.424 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-25T16:22:46.425 INFO:teuthology.orchestra.run.smithi193.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:22:46.425 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:46.425 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T16:22:46.425 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:22:46.425 INFO:teuthology.orchestra.run.smithi193.stdout:Install 2 Packages 2024-03-25T16:22:46.425 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:46.425 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 332 k 2024-03-25T16:22:46.425 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 1.5 M 2024-03-25T16:22:46.425 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T16:22:46.534 INFO:teuthology.orchestra.run.smithi193.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 628 kB/s | 69 kB 00:00 2024-03-25T16:22:47.939 INFO:teuthology.orchestra.run.smithi066.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 1.7 MB/s | 7.4 MB 00:04 2024-03-25T16:22:47.942 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:47.942 INFO:teuthology.orchestra.run.smithi066.stdout:Total 8.4 MB/s | 37 MB 00:04 2024-03-25T16:22:47.987 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T16:22:48.013 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T16:22:48.013 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T16:22:48.398 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T16:22:48.398 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T16:22:48.843 INFO:teuthology.orchestra.run.smithi193.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 109 kB/s | 264 kB 00:02 2024-03-25T16:22:48.844 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:48.844 INFO:teuthology.orchestra.run.smithi193.stdout:Total 137 kB/s | 332 kB 00:02 2024-03-25T16:22:48.845 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T16:22:48.855 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T16:22:48.855 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T16:22:48.896 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T16:22:48.898 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T16:22:49.049 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T16:22:49.234 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:22:49.262 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:22:49.278 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:22:49.278 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:22:49.278 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:22:49.278 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:49.313 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T16:22:49.396 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:22:49.426 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:22:49.528 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:22:49.571 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:22:49.756 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:22:49.786 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:22:49.871 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:22:50.023 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:22:50.364 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:22:50.364 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:50.364 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T16:22:50.364 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:50.364 INFO:teuthology.orchestra.run.smithi193.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:22:50.364 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:50.364 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T16:22:50.555 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:22:50.670 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:22:50.729 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install librados-devel 2024-03-25T16:22:51.314 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:02:53 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:22:51.461 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:22:51.462 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:22:51.462 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T16:22:51.462 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:22:51.462 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T16:22:51.463 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:22:51.463 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:51.463 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T16:22:51.463 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:22:51.463 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T16:22:51.463 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:51.463 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 127 k 2024-03-25T16:22:51.463 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 456 k 2024-03-25T16:22:51.463 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T16:22:51.973 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:22:52.633 INFO:teuthology.orchestra.run.smithi193.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 108 kB/s | 127 kB 00:01 2024-03-25T16:22:52.633 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:52.634 INFO:teuthology.orchestra.run.smithi193.stdout:Total 108 kB/s | 127 kB 00:01 2024-03-25T16:22:52.634 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T16:22:52.641 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T16:22:52.641 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T16:22:52.701 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T16:22:52.701 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T16:22:52.851 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T16:22:52.957 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:53.046 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:22:53.077 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:22:53.429 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:53.716 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:53.716 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:53.717 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T16:22:53.717 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:53.717 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:53.717 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T16:22:53.960 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install libcephfs2 2024-03-25T16:22:54.528 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:02:56 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:22:54.627 INFO:teuthology.orchestra.run.smithi193.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:22:54.683 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:22:54.684 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-03-25T16:22:54.684 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T16:22:54.739 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install libcephfs-devel 2024-03-25T16:22:55.305 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:02:57 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:22:55.455 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:22:55.456 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:22:55.456 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T16:22:55.456 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:22:55.456 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T16:22:55.456 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:22:55.456 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:55.456 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T16:22:55.457 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:22:55.457 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T16:22:55.457 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:55.457 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 31 k 2024-03-25T16:22:55.457 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 140 k 2024-03-25T16:22:55.457 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T16:22:55.558 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:22:55.558 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:22:55.558 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:22:55.558 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:22:55.558 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:22:55.558 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:22:55.558 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:22:55.559 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:22:55.559 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:22:55.559 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:22:55.559 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:22:55.559 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:22:56.005 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:22:56.005 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:22:56.005 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T16:22:56.005 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:56.005 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:22:56.005 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:22:56.005 INFO:teuthology.orchestra.run.smithi066.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:22:56.005 INFO:teuthology.orchestra.run.smithi066.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:22:56.005 INFO:teuthology.orchestra.run.smithi066.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:22:56.006 INFO:teuthology.orchestra.run.smithi066.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:22:56.006 INFO:teuthology.orchestra.run.smithi066.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:22:56.006 INFO:teuthology.orchestra.run.smithi066.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:22:56.006 INFO:teuthology.orchestra.run.smithi066.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:22:56.006 INFO:teuthology.orchestra.run.smithi066.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:22:56.006 INFO:teuthology.orchestra.run.smithi066.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:22:56.006 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:22:56.006 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T16:22:56.054 INFO:teuthology.orchestra.run.smithi193.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 52 kB/s | 31 kB 00:00 2024-03-25T16:22:56.055 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:56.055 INFO:teuthology.orchestra.run.smithi193.stdout:Total 52 kB/s | 31 kB 00:00 2024-03-25T16:22:56.055 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T16:22:56.058 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T16:22:56.059 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T16:22:56.093 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T16:22:56.094 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T16:22:56.214 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T16:22:56.336 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:56.388 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:22:56.736 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:56.985 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:02:58 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:22:56.998 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:56.998 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:56.998 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T16:22:56.998 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:56.998 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:22:56.998 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T16:22:57.141 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:22:57.143 INFO:teuthology.orchestra.run.smithi066.stdout:======================================================================================== 2024-03-25T16:22:57.143 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-03-25T16:22:57.143 INFO:teuthology.orchestra.run.smithi066.stdout:======================================================================================== 2024-03-25T16:22:57.143 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T16:22:57.143 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:22:57.143 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-25T16:22:57.143 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:22:57.143 INFO:teuthology.orchestra.run.smithi066.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:22:57.143 INFO:teuthology.orchestra.run.smithi066.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:22:57.144 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:22:57.144 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:22:57.144 INFO:teuthology.orchestra.run.smithi066.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:22:57.144 INFO:teuthology.orchestra.run.smithi066.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:22:57.144 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:22:57.144 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:22:57.144 INFO:teuthology.orchestra.run.smithi066.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:22:57.144 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:22:57.144 INFO:teuthology.orchestra.run.smithi066.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:22:57.144 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:22:57.144 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T16:22:57.144 INFO:teuthology.orchestra.run.smithi066.stdout:======================================================================================== 2024-03-25T16:22:57.144 INFO:teuthology.orchestra.run.smithi066.stdout:Install 13 Packages 2024-03-25T16:22:57.144 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:22:57.145 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 2.0 M 2024-03-25T16:22:57.145 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 26 M 2024-03-25T16:22:57.145 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T16:22:57.164 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install librados2 2024-03-25T16:22:57.469 INFO:teuthology.orchestra.run.smithi066.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 173 kB/s | 24 kB 00:00 2024-03-25T16:22:57.494 INFO:teuthology.orchestra.run.smithi066.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 109 kB/s | 18 kB 00:00 2024-03-25T16:22:57.536 INFO:teuthology.orchestra.run.smithi066.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-25T16:22:57.562 INFO:teuthology.orchestra.run.smithi066.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-25T16:22:57.586 INFO:teuthology.orchestra.run.smithi066.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T16:22:57.611 INFO:teuthology.orchestra.run.smithi066.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 857 kB/s | 43 kB 00:00 2024-03-25T16:22:57.679 INFO:teuthology.orchestra.run.smithi066.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 224 kB/s | 15 kB 00:00 2024-03-25T16:22:57.704 INFO:teuthology.orchestra.run.smithi066.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 275 kB/s | 32 kB 00:00 2024-03-25T16:22:57.741 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:02:59 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:22:57.837 INFO:teuthology.orchestra.run.smithi066.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 1.2 MB/s | 203 kB 00:00 2024-03-25T16:22:57.838 INFO:teuthology.orchestra.run.smithi193.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:22:57.862 INFO:teuthology.orchestra.run.smithi066.stdout:(10/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 98 kB/s | 52 kB 00:00 2024-03-25T16:22:57.888 INFO:teuthology.orchestra.run.smithi066.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-03-25T16:22:57.889 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:22:57.891 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-03-25T16:22:57.891 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T16:22:57.940 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install librbd1 2024-03-25T16:22:57.996 INFO:teuthology.orchestra.run.smithi066.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 671 kB/s | 90 kB 00:00 2024-03-25T16:22:58.038 INFO:teuthology.orchestra.run.smithi066.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.1 MB/s | 1.0 MB 00:00 2024-03-25T16:22:58.040 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:58.040 INFO:teuthology.orchestra.run.smithi066.stdout:Total 2.3 MB/s | 2.0 MB 00:00 2024-03-25T16:22:58.041 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T16:22:58.055 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T16:22:58.056 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T16:22:58.157 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T16:22:58.158 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T16:22:58.505 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:03:00 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:22:58.602 INFO:teuthology.orchestra.run.smithi193.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:22:58.653 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:22:58.654 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-03-25T16:22:58.654 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T16:22:58.687 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T16:22:58.706 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install python3-rados 2024-03-25T16:22:58.860 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:22:58.915 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:22:58.979 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:22:59.016 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:22:59.057 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:22:59.162 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:22:59.264 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:03:01 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:22:59.271 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:22:59.326 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:22:59.360 INFO:teuthology.orchestra.run.smithi193.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:22:59.411 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:22:59.412 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-03-25T16:22:59.412 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T16:22:59.463 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install python3-rgw 2024-03-25T16:23:00.025 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:03:02 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:23:00.088 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:23:00.123 INFO:teuthology.orchestra.run.smithi193.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:00.139 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:23:00.172 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:23:00.172 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:23:00.173 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-03-25T16:23:00.173 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T16:23:00.221 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:23:00.225 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install python3-cephfs 2024-03-25T16:23:00.250 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:23:00.793 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:03:02 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:23:00.890 INFO:teuthology.orchestra.run.smithi193.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:00.941 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:23:00.942 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-03-25T16:23:00.946 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T16:23:00.986 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install python3-rbd 2024-03-25T16:23:01.052 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:23:01.053 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:23:01.053 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:23:01.053 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:23:01.053 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:23:01.053 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:23:01.053 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:23:01.053 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:23:01.053 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:23:01.054 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:23:01.054 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:23:01.054 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:23:01.054 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:23:01.413 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:23:01.413 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:01.413 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T16:23:01.413 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:01.413 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:23:01.413 INFO:teuthology.orchestra.run.smithi066.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:23:01.413 INFO:teuthology.orchestra.run.smithi066.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:23:01.413 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:23:01.414 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:23:01.414 INFO:teuthology.orchestra.run.smithi066.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:23:01.414 INFO:teuthology.orchestra.run.smithi066.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:23:01.414 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:23:01.414 INFO:teuthology.orchestra.run.smithi066.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:23:01.414 INFO:teuthology.orchestra.run.smithi066.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:23:01.414 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:23:01.414 INFO:teuthology.orchestra.run.smithi066.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:23:01.414 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:01.414 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T16:23:01.565 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:03:03 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:23:01.636 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:23:01.669 INFO:teuthology.orchestra.run.smithi193.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:01.725 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:23:01.726 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-03-25T16:23:01.726 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T16:23:01.784 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install rbd-fuse 2024-03-25T16:23:02.227 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:04 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:23:02.379 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:03:04 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:23:02.383 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:23:02.384 INFO:teuthology.orchestra.run.smithi066.stdout:======================================================================================== 2024-03-25T16:23:02.384 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-03-25T16:23:02.384 INFO:teuthology.orchestra.run.smithi066.stdout:======================================================================================== 2024-03-25T16:23:02.384 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T16:23:02.384 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:23:02.384 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-25T16:23:02.385 INFO:teuthology.orchestra.run.smithi066.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:23:02.385 INFO:teuthology.orchestra.run.smithi066.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:23:02.385 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:23:02.385 INFO:teuthology.orchestra.run.smithi066.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:23:02.385 INFO:teuthology.orchestra.run.smithi066.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:23:02.385 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:02.385 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T16:23:02.385 INFO:teuthology.orchestra.run.smithi066.stdout:======================================================================================== 2024-03-25T16:23:02.385 INFO:teuthology.orchestra.run.smithi066.stdout:Install 6 Packages 2024-03-25T16:23:02.385 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:02.386 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 6.8 M 2024-03-25T16:23:02.386 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 33 M 2024-03-25T16:23:02.386 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T16:23:02.535 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:23:02.536 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:23:02.536 INFO:teuthology.orchestra.run.smithi193.stdout: Package Architecture Version Repository Size 2024-03-25T16:23:02.536 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:23:02.536 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T16:23:02.536 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:23:02.536 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:23:02.536 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T16:23:02.536 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:23:02.536 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T16:23:02.537 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:23:02.537 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 85 k 2024-03-25T16:23:02.537 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 231 k 2024-03-25T16:23:02.537 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T16:23:02.815 INFO:teuthology.orchestra.run.smithi066.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-25T16:23:03.140 INFO:teuthology.orchestra.run.smithi066.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.8 MB/s | 548 kB 00:00 2024-03-25T16:23:03.141 INFO:teuthology.orchestra.run.smithi193.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 141 kB/s | 85 kB 00:00 2024-03-25T16:23:03.141 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:03.141 INFO:teuthology.orchestra.run.smithi193.stdout:Total 141 kB/s | 85 kB 00:00 2024-03-25T16:23:03.141 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T16:23:03.148 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T16:23:03.148 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T16:23:03.191 INFO:teuthology.orchestra.run.smithi066.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-03-25T16:23:03.202 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T16:23:03.203 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T16:23:03.240 INFO:teuthology.orchestra.run.smithi066.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-03-25T16:23:03.324 INFO:teuthology.orchestra.run.smithi066.stdout:(5/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 174 kB/s | 138 kB 00:00 2024-03-25T16:23:03.337 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T16:23:03.439 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:03.549 INFO:teuthology.orchestra.run.smithi066.stdout:(6/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 244 kB/s | 248 kB 00:01 2024-03-25T16:23:03.551 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:03.551 INFO:teuthology.orchestra.run.smithi066.stdout:Total 5.8 MB/s | 6.8 MB 00:01 2024-03-25T16:23:03.565 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T16:23:03.577 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T16:23:03.577 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T16:23:03.674 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T16:23:03.675 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T16:23:03.845 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:03.980 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T16:23:04.129 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:04.130 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:23:04.130 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T16:23:04.130 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:04.130 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:23:04.130 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T16:23:04.137 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:23:04.193 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:23:04.410 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install rbd-mirror 2024-03-25T16:23:04.766 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:23:04.856 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:23:04.926 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:23:04.955 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:23:04.982 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:03:06 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:23:05.138 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:23:05.139 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:23:05.139 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T16:23:05.139 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:23:05.139 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T16:23:05.139 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:23:05.139 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:23:05.139 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T16:23:05.139 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:23:05.139 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T16:23:05.139 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:23:05.139 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 3.1 M 2024-03-25T16:23:05.140 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 13 M 2024-03-25T16:23:05.140 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T16:23:05.772 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:23:05.772 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:23:05.772 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:23:05.772 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:23:05.772 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:23:05.772 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:23:06.070 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:23:06.070 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:06.070 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T16:23:06.070 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:06.071 INFO:teuthology.orchestra.run.smithi066.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:23:06.071 INFO:teuthology.orchestra.run.smithi066.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:23:06.071 INFO:teuthology.orchestra.run.smithi066.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:23:06.071 INFO:teuthology.orchestra.run.smithi066.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:23:06.071 INFO:teuthology.orchestra.run.smithi066.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:23:06.071 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:06.071 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T16:23:06.315 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-fuse 2024-03-25T16:23:06.912 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:08 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:23:07.067 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:23:07.068 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:23:07.068 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-03-25T16:23:07.068 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:23:07.068 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T16:23:07.068 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:23:07.068 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-25T16:23:07.068 INFO:teuthology.orchestra.run.smithi066.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:23:07.068 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:07.068 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T16:23:07.068 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:23:07.068 INFO:teuthology.orchestra.run.smithi066.stdout:Install 2 Packages 2024-03-25T16:23:07.068 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:07.069 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 964 k 2024-03-25T16:23:07.069 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 2.8 M 2024-03-25T16:23:07.069 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T16:23:07.192 INFO:teuthology.orchestra.run.smithi066.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 639 kB/s | 79 kB 00:00 2024-03-25T16:23:08.134 INFO:teuthology.orchestra.run.smithi066.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 831 kB/s | 885 kB 00:01 2024-03-25T16:23:08.136 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:08.136 INFO:teuthology.orchestra.run.smithi066.stdout:Total 903 kB/s | 964 kB 00:01 2024-03-25T16:23:08.136 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T16:23:08.161 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T16:23:08.161 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T16:23:08.243 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T16:23:08.243 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T16:23:08.439 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T16:23:08.582 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:23:08.653 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:23:09.398 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:23:09.399 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:23:09.711 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:23:09.711 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:09.711 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T16:23:09.711 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:23:09.711 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:09.711 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T16:23:09.971 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-volume 2024-03-25T16:23:10.567 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:12 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:23:10.725 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:23:10.725 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:23:10.725 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repository Size 2024-03-25T16:23:10.725 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:23:10.725 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T16:23:10.725 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:23:10.726 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-25T16:23:10.726 INFO:teuthology.orchestra.run.smithi066.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:23:10.726 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:10.726 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T16:23:10.726 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:23:10.726 INFO:teuthology.orchestra.run.smithi066.stdout:Install 2 Packages 2024-03-25T16:23:10.726 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:10.726 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 332 k 2024-03-25T16:23:10.727 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 1.5 M 2024-03-25T16:23:10.727 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T16:23:10.838 INFO:teuthology.orchestra.run.smithi066.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 618 kB/s | 69 kB 00:00 2024-03-25T16:23:11.622 INFO:teuthology.orchestra.run.smithi066.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 295 kB/s | 264 kB 00:00 2024-03-25T16:23:11.623 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:11.624 INFO:teuthology.orchestra.run.smithi066.stdout:Total 370 kB/s | 332 kB 00:00 2024-03-25T16:23:11.624 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T16:23:11.634 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T16:23:11.634 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T16:23:11.676 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T16:23:11.676 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T16:23:11.834 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T16:23:12.028 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:23:12.066 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:23:12.083 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:23:12.083 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:23:12.083 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:23:12.083 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:12.838 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:23:13.146 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:23:13.147 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:13.147 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T16:23:13.147 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:13.147 INFO:teuthology.orchestra.run.smithi066.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:23:13.147 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:13.147 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T16:23:13.377 INFO:teuthology.orchestra.run.smithi193.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 384 kB/s | 3.1 MB 00:08 2024-03-25T16:23:13.378 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:13.378 INFO:teuthology.orchestra.run.smithi193.stdout:Total 384 kB/s | 3.1 MB 00:08 2024-03-25T16:23:13.378 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T16:23:13.390 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T16:23:13.390 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T16:23:13.464 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install librados-devel 2024-03-25T16:23:13.467 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T16:23:13.467 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T16:23:13.891 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T16:23:13.944 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:13.969 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:13.969 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:23:13.969 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:23:13.969 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:23:13.969 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:23:13.969 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:23:14.064 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:16 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:23:14.220 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:23:14.220 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:23:14.221 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-25T16:23:14.221 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:23:14.221 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T16:23:14.221 INFO:teuthology.orchestra.run.smithi066.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:23:14.221 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:14.221 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T16:23:14.221 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:23:14.221 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-25T16:23:14.221 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:14.221 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 127 k 2024-03-25T16:23:14.221 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 456 k 2024-03-25T16:23:14.221 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T16:23:14.885 INFO:teuthology.orchestra.run.smithi066.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 191 kB/s | 127 kB 00:00 2024-03-25T16:23:14.886 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:14.886 INFO:teuthology.orchestra.run.smithi066.stdout:Total 190 kB/s | 127 kB 00:00 2024-03-25T16:23:14.886 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T16:23:14.893 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T16:23:14.893 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T16:23:14.954 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T16:23:14.955 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T16:23:15.103 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T16:23:15.110 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:15.110 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:23:15.110 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T16:23:15.110 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:15.110 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:23:15.110 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T16:23:15.212 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:15.363 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install rbd-nbd 2024-03-25T16:23:15.657 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:15.944 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:15.944 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:15.944 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T16:23:15.944 INFO:teuthology.orchestra.run.smithi066.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:15.944 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:15.944 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T16:23:15.960 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:03:17 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:23:16.108 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:23:16.109 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:23:16.109 INFO:teuthology.orchestra.run.smithi193.stdout: Package Architecture Version Repository Size 2024-03-25T16:23:16.109 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:23:16.109 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T16:23:16.109 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:23:16.110 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:23:16.110 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T16:23:16.110 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:23:16.110 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T16:23:16.110 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:23:16.110 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 171 k 2024-03-25T16:23:16.110 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 494 k 2024-03-25T16:23:16.110 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T16:23:16.167 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install libcephfs2 2024-03-25T16:23:16.759 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:18 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:23:16.859 INFO:teuthology.orchestra.run.smithi066.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:16.915 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:23:16.916 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-03-25T16:23:16.916 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T16:23:16.980 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install libcephfs-devel 2024-03-25T16:23:17.576 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:19 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:23:17.733 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:23:17.734 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:23:17.734 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-25T16:23:17.734 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:23:17.734 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T16:23:17.734 INFO:teuthology.orchestra.run.smithi066.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:23:17.734 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:17.734 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T16:23:17.734 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:23:17.735 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-25T16:23:17.735 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:17.735 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 31 k 2024-03-25T16:23:17.735 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 140 k 2024-03-25T16:23:17.735 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T16:23:17.847 INFO:teuthology.orchestra.run.smithi193.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 99 kB/s | 171 kB 00:01 2024-03-25T16:23:17.848 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:17.848 INFO:teuthology.orchestra.run.smithi193.stdout:Total 99 kB/s | 171 kB 00:01 2024-03-25T16:23:17.848 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T16:23:17.858 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T16:23:17.858 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T16:23:17.918 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T16:23:17.919 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T16:23:18.071 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T16:23:18.147 INFO:teuthology.orchestra.run.smithi066.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 76 kB/s | 31 kB 00:00 2024-03-25T16:23:18.147 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:18.148 INFO:teuthology.orchestra.run.smithi066.stdout:Total 75 kB/s | 31 kB 00:00 2024-03-25T16:23:18.148 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T16:23:18.151 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T16:23:18.151 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T16:23:18.178 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:18.179 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T16:23:18.179 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T16:23:18.308 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T16:23:18.415 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:18.621 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:18.812 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:18.900 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:18.900 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:23:18.900 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T16:23:18.900 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:18.900 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:23:18.900 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T16:23:19.094 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:19.094 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:19.094 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T16:23:19.094 INFO:teuthology.orchestra.run.smithi066.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:19.094 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:19.094 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T16:23:19.142 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:23:19.249 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install librados2 2024-03-25T16:23:19.712 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:03:21 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:23:19.845 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:21 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:23:19.867 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:23:19.868 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:23:19.868 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T16:23:19.868 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:23:19.868 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T16:23:19.868 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:23:19.868 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-25T16:23:19.868 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:23:19.869 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:23:19.869 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T16:23:19.869 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:23:19.869 INFO:teuthology.orchestra.run.smithi193.stdout:Install 2 Packages 2024-03-25T16:23:19.869 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:23:19.869 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 91 M 2024-03-25T16:23:19.869 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 498 M 2024-03-25T16:23:19.869 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T16:23:19.946 INFO:teuthology.orchestra.run.smithi066.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:20.000 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:23:20.001 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-03-25T16:23:20.001 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T16:23:20.045 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install librbd1 2024-03-25T16:23:20.645 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:22 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:23:20.746 INFO:teuthology.orchestra.run.smithi066.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:20.801 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:23:20.802 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-03-25T16:23:20.802 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T16:23:20.855 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install python3-rados 2024-03-25T16:23:21.445 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:23 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:23:21.544 INFO:teuthology.orchestra.run.smithi066.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:21.598 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:23:21.600 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-03-25T16:23:21.600 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T16:23:21.644 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install python3-rgw 2024-03-25T16:23:22.237 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:24 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:23:22.336 INFO:teuthology.orchestra.run.smithi066.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:22.391 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:23:22.392 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-03-25T16:23:22.392 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T16:23:22.436 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install python3-cephfs 2024-03-25T16:23:23.032 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:25 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:23:23.131 INFO:teuthology.orchestra.run.smithi066.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:23.188 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:23:23.189 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-03-25T16:23:23.189 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T16:23:23.240 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install python3-rbd 2024-03-25T16:23:23.836 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:25 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:23:23.935 INFO:teuthology.orchestra.run.smithi066.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:23.990 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:23:23.991 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-03-25T16:23:23.991 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T16:23:24.037 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install rbd-fuse 2024-03-25T16:23:24.652 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:26 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:23:24.810 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:23:24.811 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:23:24.811 INFO:teuthology.orchestra.run.smithi066.stdout: Package Architecture Version Repository Size 2024-03-25T16:23:24.811 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:23:24.811 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T16:23:24.811 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:23:24.811 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:24.811 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T16:23:24.811 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:23:24.812 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-25T16:23:24.812 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:24.812 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 85 k 2024-03-25T16:23:24.812 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 231 k 2024-03-25T16:23:24.812 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T16:23:25.513 INFO:teuthology.orchestra.run.smithi066.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 122 kB/s | 85 kB 00:00 2024-03-25T16:23:25.513 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:25.514 INFO:teuthology.orchestra.run.smithi066.stdout:Total 121 kB/s | 85 kB 00:00 2024-03-25T16:23:25.514 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T16:23:25.522 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T16:23:25.522 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T16:23:25.581 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T16:23:25.581 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T16:23:25.734 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T16:23:25.835 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:26.250 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:26.507 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:26.508 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:26.508 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T16:23:26.508 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:26.508 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:26.508 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T16:23:26.752 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install rbd-mirror 2024-03-25T16:23:27.349 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:29 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:23:27.502 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:23:27.503 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:23:27.503 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-25T16:23:27.503 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:23:27.504 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T16:23:27.504 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:23:27.504 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:27.504 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T16:23:27.504 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:23:27.504 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-25T16:23:27.504 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:27.504 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 3.1 M 2024-03-25T16:23:27.504 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 13 M 2024-03-25T16:23:27.504 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T16:23:32.727 INFO:teuthology.orchestra.run.smithi066.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 605 kB/s | 3.1 MB 00:05 2024-03-25T16:23:32.728 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:32.728 INFO:teuthology.orchestra.run.smithi066.stdout:Total 605 kB/s | 3.1 MB 00:05 2024-03-25T16:23:32.728 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T16:23:32.741 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T16:23:32.741 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T16:23:32.822 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T16:23:32.822 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T16:23:33.271 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T16:23:33.336 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:33.362 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:33.363 INFO:teuthology.orchestra.run.smithi066.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:23:33.363 INFO:teuthology.orchestra.run.smithi066.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:23:33.363 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:23:33.363 INFO:teuthology.orchestra.run.smithi066.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:23:33.363 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:34.542 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:34.542 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:34.542 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T16:23:34.543 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:34.543 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:34.543 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T16:23:34.781 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install rbd-nbd 2024-03-25T16:23:35.412 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:37 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:23:35.569 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:23:35.570 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:23:35.570 INFO:teuthology.orchestra.run.smithi066.stdout: Package Architecture Version Repository Size 2024-03-25T16:23:35.570 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:23:35.570 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T16:23:35.570 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:23:35.570 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:35.570 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T16:23:35.570 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:23:35.570 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-25T16:23:35.570 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:35.571 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 171 k 2024-03-25T16:23:35.571 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 494 k 2024-03-25T16:23:35.571 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T16:23:36.230 INFO:teuthology.orchestra.run.smithi066.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 260 kB/s | 171 kB 00:00 2024-03-25T16:23:36.230 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:36.231 INFO:teuthology.orchestra.run.smithi066.stdout:Total 260 kB/s | 171 kB 00:00 2024-03-25T16:23:36.231 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T16:23:36.240 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T16:23:36.240 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T16:23:36.301 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T16:23:36.302 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T16:23:36.459 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T16:23:36.556 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:37.035 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:37.341 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:37.341 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:37.341 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T16:23:37.342 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:37.342 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:37.342 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T16:23:37.595 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:23:38.194 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:03:40 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:23:38.348 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:23:38.349 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:23:38.349 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-25T16:23:38.349 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:23:38.349 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T16:23:38.350 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:23:38.350 INFO:teuthology.orchestra.run.smithi066.stdout:Installing dependencies: 2024-03-25T16:23:38.350 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:23:38.350 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:38.350 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T16:23:38.350 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:23:38.350 INFO:teuthology.orchestra.run.smithi066.stdout:Install 2 Packages 2024-03-25T16:23:38.350 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:23:38.350 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 91 M 2024-03-25T16:23:38.350 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 498 M 2024-03-25T16:23:38.351 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T16:24:01.963 INFO:teuthology.orchestra.run.smithi193.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 546 kB/s | 22 MB 00:42 2024-03-25T16:24:07.865 INFO:teuthology.orchestra.run.smithi066.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 779 kB/s | 22 MB 00:29 2024-03-25T16:25:20.951 INFO:teuthology.orchestra.run.smithi193.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 576 kB/s | 68 MB 02:01 2024-03-25T16:25:20.952 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:20.953 INFO:teuthology.orchestra.run.smithi193.stdout:Total 766 kB/s | 91 MB 02:01 2024-03-25T16:25:20.953 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T16:25:20.957 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T16:25:20.957 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T16:25:21.547 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T16:25:21.548 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T16:25:23.482 INFO:teuthology.orchestra.run.smithi066.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 664 kB/s | 68 MB 01:45 2024-03-25T16:25:23.483 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:23.484 INFO:teuthology.orchestra.run.smithi066.stdout:Total 882 kB/s | 91 MB 01:45 2024-03-25T16:25:23.484 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T16:25:23.488 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T16:25:23.488 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T16:25:24.096 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T16:25:24.097 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T16:25:24.662 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T16:25:27.367 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T16:25:31.788 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:25:31.868 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:25:34.888 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:25:34.960 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:25:35.133 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:25:35.133 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:25:35.393 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:25:35.393 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:25:35.393 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T16:25:35.393 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:35.393 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:35.393 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:25:35.393 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T16:25:35.611 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:25:36.187 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:05:38 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:25:36.345 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:25:36.346 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:25:36.346 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T16:25:36.346 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:25:36.346 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T16:25:36.346 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:25:36.346 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:25:36.346 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T16:25:36.346 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:25:36.346 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T16:25:36.346 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:25:36.350 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 416 M 2024-03-25T16:25:36.350 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 1.7 G 2024-03-25T16:25:36.350 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T16:25:38.398 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:25:38.399 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:25:38.666 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:25:38.666 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:25:38.666 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T16:25:38.666 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:38.667 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:38.667 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:25:38.667 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T16:25:39.017 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:25:39.620 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:05:41 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:25:39.782 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:25:39.783 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:25:39.783 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-25T16:25:39.783 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:25:39.783 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T16:25:39.783 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:25:39.784 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:25:39.784 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T16:25:39.784 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:25:39.784 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-25T16:25:39.784 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:25:39.784 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 416 M 2024-03-25T16:25:39.784 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 1.7 G 2024-03-25T16:25:39.784 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T16:49:13.095 INFO:teuthology.orchestra.run.smithi066.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 301 kB/s | 416 MB 23:33 2024-03-25T16:49:13.118 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:49:13.118 INFO:teuthology.orchestra.run.smithi066.stdout:Total 301 kB/s | 416 MB 23:33 2024-03-25T16:49:13.118 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T16:49:13.118 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T16:49:13.118 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T16:49:15.692 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T16:49:15.692 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T16:49:22.552 INFO:teuthology.orchestra.run.smithi193.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 299 kB/s | 416 MB 23:46 2024-03-25T16:49:22.552 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:49:22.553 INFO:teuthology.orchestra.run.smithi193.stdout:Total 299 kB/s | 416 MB 23:46 2024-03-25T16:49:22.553 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T16:49:22.556 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T16:49:22.557 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T16:49:25.019 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T16:49:25.020 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T16:50:00.338 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T16:50:00.433 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:50:06.980 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T16:50:07.063 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:50:07.308 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:50:07.582 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:50:07.582 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:50:07.582 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T16:50:07.582 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:50:07.582 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:50:07.582 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T16:50:07.841 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:50:08.437 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:30:10 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:50:08.591 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:50:08.592 INFO:teuthology.orchestra.run.smithi066.stdout:======================================================================================== 2024-03-25T16:50:08.592 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-25T16:50:08.593 INFO:teuthology.orchestra.run.smithi066.stdout:======================================================================================== 2024-03-25T16:50:08.593 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T16:50:08.593 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:50:08.593 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:50:08.593 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T16:50:08.593 INFO:teuthology.orchestra.run.smithi066.stdout:======================================================================================== 2024-03-25T16:50:08.593 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-25T16:50:08.593 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:50:08.593 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 1.4 M 2024-03-25T16:50:08.593 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 6.4 M 2024-03-25T16:50:08.593 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T16:50:11.956 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:50:12.218 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:50:12.218 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:50:12.219 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T16:50:12.219 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:50:12.219 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:50:12.219 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T16:50:12.453 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:50:13.047 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:30:15 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:50:13.200 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:50:13.201 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================== 2024-03-25T16:50:13.201 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T16:50:13.201 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================== 2024-03-25T16:50:13.201 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T16:50:13.201 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:50:13.201 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:50:13.201 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T16:50:13.201 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================== 2024-03-25T16:50:13.202 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T16:50:13.202 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:50:13.202 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 1.4 M 2024-03-25T16:50:13.202 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 6.4 M 2024-03-25T16:50:13.202 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T16:50:14.818 INFO:teuthology.orchestra.run.smithi066.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 235 kB/s | 1.4 MB 00:06 2024-03-25T16:50:14.819 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:50:14.819 INFO:teuthology.orchestra.run.smithi066.stdout:Total 235 kB/s | 1.4 MB 00:06 2024-03-25T16:50:14.819 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T16:50:14.824 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T16:50:14.824 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T16:50:14.873 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T16:50:14.874 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T16:50:15.141 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T16:50:15.252 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:50:15.799 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:50:16.078 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:50:16.078 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:50:16.078 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T16:50:16.078 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:50:16.078 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:50:16.078 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T16:50:16.261 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:50:16.854 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:30:18 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:50:17.010 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:50:17.010 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:50:17.010 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-25T16:50:17.010 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:50:17.010 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T16:50:17.011 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:50:17.011 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:50:17.011 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T16:50:17.011 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:50:17.011 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-25T16:50:17.011 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:50:17.011 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 176 M 2024-03-25T16:50:17.011 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 753 M 2024-03-25T16:50:17.012 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T16:50:20.367 INFO:teuthology.orchestra.run.smithi193.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 204 kB/s | 1.4 MB 00:07 2024-03-25T16:50:20.368 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:50:20.368 INFO:teuthology.orchestra.run.smithi193.stdout:Total 204 kB/s | 1.4 MB 00:07 2024-03-25T16:50:20.368 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T16:50:20.372 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T16:50:20.372 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T16:50:20.416 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T16:50:20.416 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T16:50:20.686 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T16:50:20.791 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:50:21.203 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:50:21.479 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:50:21.479 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:50:21.479 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T16:50:21.479 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:50:21.479 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:50:21.479 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T16:50:21.649 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:50:22.208 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:30:24 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:50:22.355 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:50:22.355 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:50:22.355 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T16:50:22.355 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:50:22.355 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T16:50:22.356 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:50:22.356 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:50:22.356 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T16:50:22.356 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:50:22.356 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T16:50:22.356 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:50:22.356 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 176 M 2024-03-25T16:50:22.356 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 753 M 2024-03-25T16:50:22.356 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T16:57:33.184 INFO:teuthology.orchestra.run.smithi066.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 412 kB/s | 176 MB 07:16 2024-03-25T16:57:33.184 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:57:33.184 INFO:teuthology.orchestra.run.smithi066.stdout:Total 412 kB/s | 176 MB 07:16 2024-03-25T16:57:33.184 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T16:57:33.187 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T16:57:33.187 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T16:57:33.215 INFO:teuthology.orchestra.run.smithi193.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 417 kB/s | 176 MB 07:10 2024-03-25T16:57:33.215 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:57:33.216 INFO:teuthology.orchestra.run.smithi193.stdout:Total 417 kB/s | 176 MB 07:10 2024-03-25T16:57:33.216 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T16:57:33.219 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T16:57:33.219 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T16:57:34.281 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T16:57:34.281 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T16:57:34.299 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T16:57:34.299 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T16:57:52.195 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T16:57:52.294 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:57:53.421 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T16:57:53.517 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:57:56.695 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:57:56.939 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:57:56.939 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:57:56.939 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T16:57:56.939 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:57:56.939 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:57:56.939 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T16:57:57.126 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-test-debuginfo 2024-03-25T16:57:57.687 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:37:59 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:57:57.838 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T16:57:57.839 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:57:57.839 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T16:57:57.839 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:57:57.839 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T16:57:57.839 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T16:57:57.840 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:57:57.840 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T16:57:57.840 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T16:57:57.840 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T16:57:57.840 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T16:57:57.840 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 672 M 2024-03-25T16:57:57.840 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 3.1 G 2024-03-25T16:57:57.840 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T16:57:57.895 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:57:58.171 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:57:58.171 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:57:58.171 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T16:57:58.172 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:57:58.172 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:57:58.172 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T16:57:58.496 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-test-debuginfo 2024-03-25T16:57:59.094 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 0:38:01 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T16:57:59.249 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T16:57:59.250 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:57:59.250 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-25T16:57:59.250 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:57:59.250 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T16:57:59.250 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T16:57:59.251 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:57:59.251 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T16:57:59.251 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T16:57:59.251 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-25T16:57:59.251 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T16:57:59.251 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 672 M 2024-03-25T16:57:59.251 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 3.1 G 2024-03-25T16:57:59.251 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T17:27:53.660 INFO:teuthology.orchestra.run.smithi066.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 384 kB/s | 672 MB 29:54 2024-03-25T17:27:53.680 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:27:53.680 INFO:teuthology.orchestra.run.smithi066.stdout:Total 384 kB/s | 672 MB 29:54 2024-03-25T17:27:53.680 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T17:27:53.681 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T17:27:53.681 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T17:27:57.849 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T17:27:57.849 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T17:28:34.411 INFO:teuthology.orchestra.run.smithi193.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 375 kB/s | 672 MB 30:36 2024-03-25T17:28:34.412 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:28:34.412 INFO:teuthology.orchestra.run.smithi193.stdout:Total 375 kB/s | 672 MB 30:36 2024-03-25T17:28:34.412 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T17:28:34.417 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T17:28:34.417 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T17:28:38.373 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T17:28:38.373 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T17:29:13.089 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T17:29:13.196 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:29:22.505 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:29:22.764 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:29:22.764 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:29:22.764 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T17:29:22.764 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:29:22.764 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:29:22.765 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T17:29:23.043 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:29:23.637 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 1:09:25 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:29:23.738 INFO:teuthology.orchestra.run.smithi066.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:29:23.793 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T17:29:23.795 INFO:teuthology.orchestra.run.smithi066.stdout:Nothing to do. 2024-03-25T17:29:23.795 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T17:29:23.868 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:29:24.462 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 1:09:26 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:29:24.617 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T17:29:24.617 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:29:24.617 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-25T17:29:24.617 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:29:24.617 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T17:29:24.617 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:29:24.618 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:29:24.618 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T17:29:24.618 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:29:24.618 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-25T17:29:24.618 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:29:24.618 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 20 M 2024-03-25T17:29:24.618 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 103 M 2024-03-25T17:29:24.618 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T17:29:49.264 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T17:29:49.872 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:29:57.557 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:29:57.815 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:29:57.815 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:29:57.815 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T17:29:57.816 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:29:57.816 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:29:57.816 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T17:29:58.142 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:29:58.706 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:10:00 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:29:58.801 INFO:teuthology.orchestra.run.smithi193.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:29:58.854 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T17:29:58.855 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2024-03-25T17:29:58.855 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T17:29:58.909 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:29:59.470 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:10:01 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:29:59.618 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T17:29:59.619 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:29:59.619 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T17:29:59.619 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:29:59.619 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T17:29:59.619 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:29:59.619 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:29:59.619 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T17:29:59.619 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:29:59.620 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T17:29:59.620 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:29:59.620 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 20 M 2024-03-25T17:29:59.620 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 103 M 2024-03-25T17:29:59.620 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T17:30:28.448 INFO:teuthology.orchestra.run.smithi066.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 316 kB/s | 20 MB 01:03 2024-03-25T17:30:28.449 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:30:28.449 INFO:teuthology.orchestra.run.smithi066.stdout:Total 316 kB/s | 20 MB 01:03 2024-03-25T17:30:28.449 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T17:30:28.453 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T17:30:28.454 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T17:30:28.611 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T17:30:28.611 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T17:30:30.981 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T17:30:31.083 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:30:32.021 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:30:32.327 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:30:32.327 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:30:32.327 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T17:30:32.327 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:30:32.328 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:30:32.328 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T17:30:32.520 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:30:33.115 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 1:10:35 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:30:33.269 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T17:30:33.269 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:30:33.269 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-25T17:30:33.269 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:30:33.269 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T17:30:33.269 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:30:33.270 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:30:33.270 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T17:30:33.270 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:30:33.270 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-25T17:30:33.270 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:30:33.270 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 48 M 2024-03-25T17:30:33.270 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 173 M 2024-03-25T17:30:33.271 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T17:30:46.933 INFO:teuthology.orchestra.run.smithi193.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 426 kB/s | 20 MB 00:47 2024-03-25T17:30:46.934 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:30:46.934 INFO:teuthology.orchestra.run.smithi193.stdout:Total 426 kB/s | 20 MB 00:47 2024-03-25T17:30:46.935 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T17:30:46.938 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T17:30:46.938 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T17:30:47.087 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T17:30:47.087 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T17:30:49.302 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T17:30:49.414 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:30:50.356 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:30:50.612 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:30:50.613 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:30:50.613 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T17:30:50.613 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:30:50.613 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:30:50.613 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T17:30:50.779 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:30:51.348 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:10:53 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:30:51.498 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T17:30:51.498 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:30:51.499 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T17:30:51.499 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:30:51.499 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T17:30:51.499 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:30:51.499 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:30:51.499 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T17:30:51.499 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:30:51.499 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T17:30:51.499 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:30:51.499 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 48 M 2024-03-25T17:30:51.499 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 173 M 2024-03-25T17:30:51.499 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T17:32:57.919 INFO:teuthology.orchestra.run.smithi066.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 338 kB/s | 48 MB 02:24 2024-03-25T17:32:57.920 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:32:57.920 INFO:teuthology.orchestra.run.smithi066.stdout:Total 338 kB/s | 48 MB 02:24 2024-03-25T17:32:57.920 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T17:32:57.923 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T17:32:57.923 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T17:32:58.252 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T17:32:58.253 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T17:33:03.277 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T17:33:03.378 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:33:04.733 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:33:04.991 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:33:04.991 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:33:04.991 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T17:33:04.991 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:33:04.991 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:33:04.991 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T17:33:05.173 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:33:05.769 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 1:13:07 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:33:05.924 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T17:33:05.924 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:33:05.924 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-25T17:33:05.925 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:33:05.925 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T17:33:05.925 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:33:05.925 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:33:05.925 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T17:33:05.925 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:33:05.925 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-25T17:33:05.925 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:33:05.925 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 99 M 2024-03-25T17:33:05.925 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 449 M 2024-03-25T17:33:05.925 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T17:33:28.280 INFO:teuthology.orchestra.run.smithi193.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 312 kB/s | 48 MB 02:36 2024-03-25T17:33:28.280 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:33:28.281 INFO:teuthology.orchestra.run.smithi193.stdout:Total 312 kB/s | 48 MB 02:36 2024-03-25T17:33:28.281 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T17:33:28.284 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T17:33:28.284 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T17:33:28.595 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T17:33:28.596 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T17:33:33.318 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T17:33:33.477 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:33:34.817 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:33:35.075 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:33:35.075 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:33:35.075 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T17:33:35.075 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:33:35.075 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:33:35.076 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T17:33:35.261 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:33:35.833 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:13:37 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:33:35.981 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T17:33:35.982 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:33:35.982 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T17:33:35.982 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:33:35.982 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T17:33:35.982 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:33:35.982 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:33:35.982 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T17:33:35.983 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:33:35.983 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T17:33:35.983 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:33:35.983 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 99 M 2024-03-25T17:33:35.983 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 449 M 2024-03-25T17:33:35.983 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T17:37:43.922 INFO:teuthology.orchestra.run.smithi193.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 407 kB/s | 99 MB 04:07 2024-03-25T17:37:43.922 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:37:43.922 INFO:teuthology.orchestra.run.smithi193.stdout:Total 407 kB/s | 99 MB 04:07 2024-03-25T17:37:43.923 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T17:37:43.925 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T17:37:43.925 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T17:37:44.531 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T17:37:44.531 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T17:37:54.766 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T17:37:54.867 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:37:57.831 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:37:58.083 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:37:58.083 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:37:58.083 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T17:37:58.083 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:37:58.083 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:37:58.083 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T17:37:58.253 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:37:58.820 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:18:00 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:37:58.971 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T17:37:58.971 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:37:58.972 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T17:37:58.972 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:37:58.972 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T17:37:58.972 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:37:58.972 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:37:58.972 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T17:37:58.972 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:37:58.972 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T17:37:58.972 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:37:58.973 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 287 M 2024-03-25T17:37:58.973 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 1.2 G 2024-03-25T17:37:58.973 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T17:37:59.899 INFO:teuthology.orchestra.run.smithi066.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 343 kB/s | 99 MB 04:53 2024-03-25T17:37:59.899 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:37:59.900 INFO:teuthology.orchestra.run.smithi066.stdout:Total 343 kB/s | 99 MB 04:53 2024-03-25T17:37:59.900 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T17:37:59.904 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T17:37:59.904 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T17:38:00.544 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T17:38:00.545 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T17:38:11.427 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T17:38:11.566 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:38:14.386 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:38:14.677 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:38:14.677 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:38:14.677 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T17:38:14.677 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:38:14.677 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:38:14.677 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T17:38:14.854 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:38:15.449 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 1:18:17 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:38:15.606 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T17:38:15.607 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:38:15.607 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-25T17:38:15.607 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:38:15.607 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T17:38:15.607 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:38:15.607 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:38:15.607 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T17:38:15.607 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:38:15.607 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-25T17:38:15.607 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:38:15.607 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 287 M 2024-03-25T17:38:15.607 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 1.2 G 2024-03-25T17:38:15.607 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T17:51:37.812 INFO:teuthology.orchestra.run.smithi193.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 359 kB/s | 287 MB 13:38 2024-03-25T17:51:37.812 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:37.812 INFO:teuthology.orchestra.run.smithi193.stdout:Total 359 kB/s | 287 MB 13:38 2024-03-25T17:51:37.813 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T17:51:37.816 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T17:51:37.816 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T17:51:39.519 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T17:51:39.519 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T17:52:08.816 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T17:52:08.915 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:52:16.379 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:52:16.670 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:52:16.671 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:52:16.671 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T17:52:16.671 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:52:16.671 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:52:16.671 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T17:52:16.881 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T17:52:17.444 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:32:19 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:52:17.593 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T17:52:17.594 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:52:17.594 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T17:52:17.594 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:52:17.594 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T17:52:17.595 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T17:52:17.595 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:52:17.595 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T17:52:17.595 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:52:17.595 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T17:52:17.595 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:52:17.595 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 5.9 M 2024-03-25T17:52:17.595 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 28 M 2024-03-25T17:52:17.595 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T17:52:37.049 INFO:teuthology.orchestra.run.smithi193.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 313 kB/s | 5.9 MB 00:19 2024-03-25T17:52:37.050 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:37.050 INFO:teuthology.orchestra.run.smithi193.stdout:Total 313 kB/s | 5.9 MB 00:19 2024-03-25T17:52:37.050 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T17:52:37.054 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T17:52:37.054 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T17:52:37.125 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T17:52:37.125 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T17:52:37.859 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T17:52:37.955 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:52:38.563 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:52:38.814 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:52:38.814 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:52:38.815 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T17:52:38.815 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:52:38.815 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:52:38.815 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T17:52:39.005 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install librados-devel-debuginfo 2024-03-25T17:52:39.568 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:32:41 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:52:39.717 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T17:52:39.717 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:52:39.717 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T17:52:39.718 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:52:39.718 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T17:52:39.718 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T17:52:39.718 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:52:39.718 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T17:52:39.718 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:52:39.718 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T17:52:39.718 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:52:39.718 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 109 k 2024-03-25T17:52:39.718 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 440 k 2024-03-25T17:52:39.718 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T17:52:40.527 INFO:teuthology.orchestra.run.smithi193.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 134 kB/s | 109 kB 00:00 2024-03-25T17:52:40.528 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:40.528 INFO:teuthology.orchestra.run.smithi193.stdout:Total 134 kB/s | 109 kB 00:00 2024-03-25T17:52:40.528 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T17:52:40.532 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T17:52:40.532 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T17:52:40.566 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T17:52:40.567 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T17:52:40.682 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T17:52:40.783 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:52:41.241 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:52:41.488 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:52:41.488 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:52:41.488 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T17:52:41.488 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:52:41.488 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:52:41.488 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T17:52:41.649 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T17:52:42.214 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:32:44 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:52:42.361 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T17:52:42.362 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:52:42.362 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T17:52:42.362 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:52:42.362 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T17:52:42.362 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T17:52:42.362 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:52:42.362 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T17:52:42.362 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:52:42.363 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T17:52:42.363 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:52:42.363 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 5.4 M 2024-03-25T17:52:42.363 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 25 M 2024-03-25T17:52:42.363 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T17:52:53.770 INFO:teuthology.orchestra.run.smithi193.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 480 kB/s | 5.4 MB 00:11 2024-03-25T17:52:53.771 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:53.771 INFO:teuthology.orchestra.run.smithi193.stdout:Total 480 kB/s | 5.4 MB 00:11 2024-03-25T17:52:53.772 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T17:52:53.776 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T17:52:53.776 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T17:52:53.844 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T17:52:53.844 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T17:52:54.510 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T17:52:54.618 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:52:55.154 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:52:55.396 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:52:55.397 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:52:55.397 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T17:52:55.397 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:52:55.397 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:52:55.397 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T17:52:55.578 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install librados2-debuginfo 2024-03-25T17:52:56.141 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:32:58 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:52:56.289 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T17:52:56.290 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:52:56.290 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T17:52:56.290 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:52:56.290 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T17:52:56.290 INFO:teuthology.orchestra.run.smithi193.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T17:52:56.290 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:52:56.290 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T17:52:56.290 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:52:56.291 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T17:52:56.291 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:52:56.291 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 44 M 2024-03-25T17:52:56.291 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 253 M 2024-03-25T17:52:56.291 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T17:53:47.192 INFO:teuthology.orchestra.run.smithi066.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 315 kB/s | 287 MB 15:31 2024-03-25T17:53:47.193 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:53:47.193 INFO:teuthology.orchestra.run.smithi066.stdout:Total 315 kB/s | 287 MB 15:31 2024-03-25T17:53:47.193 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T17:53:47.197 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T17:53:47.197 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T17:53:48.993 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T17:53:48.993 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T17:54:20.068 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T17:54:20.174 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:54:27.429 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:54:27.721 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:54:27.721 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:54:27.721 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T17:54:27.721 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:54:27.721 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:54:27.721 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T17:54:27.948 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T17:54:28.543 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 1:34:30 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:54:28.699 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T17:54:28.700 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:54:28.700 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-25T17:54:28.700 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:54:28.700 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T17:54:28.700 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T17:54:28.700 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:54:28.700 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T17:54:28.700 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:54:28.700 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-25T17:54:28.700 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:54:28.701 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 5.9 M 2024-03-25T17:54:28.701 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 28 M 2024-03-25T17:54:28.701 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T17:54:46.198 INFO:teuthology.orchestra.run.smithi066.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 348 kB/s | 5.9 MB 00:17 2024-03-25T17:54:46.199 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:54:46.200 INFO:teuthology.orchestra.run.smithi066.stdout:Total 348 kB/s | 5.9 MB 00:17 2024-03-25T17:54:46.200 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T17:54:46.204 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T17:54:46.204 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T17:54:46.281 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T17:54:46.282 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T17:54:47.065 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T17:54:47.184 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:54:47.718 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:54:47.986 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:54:47.987 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:54:47.987 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T17:54:47.987 INFO:teuthology.orchestra.run.smithi066.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:54:47.987 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:54:47.987 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T17:54:48.210 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install librados-devel-debuginfo 2024-03-25T17:54:48.803 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 1:34:50 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:54:48.959 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T17:54:48.960 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:54:48.960 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-25T17:54:48.960 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:54:48.960 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T17:54:48.960 INFO:teuthology.orchestra.run.smithi066.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T17:54:48.960 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:54:48.960 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T17:54:48.960 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:54:48.960 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-25T17:54:48.960 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:54:48.960 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 109 k 2024-03-25T17:54:48.960 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 440 k 2024-03-25T17:54:48.960 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T17:54:49.834 INFO:teuthology.orchestra.run.smithi066.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 124 kB/s | 109 kB 00:00 2024-03-25T17:54:49.835 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:54:49.835 INFO:teuthology.orchestra.run.smithi066.stdout:Total 124 kB/s | 109 kB 00:00 2024-03-25T17:54:49.835 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T17:54:49.839 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T17:54:49.839 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T17:54:49.880 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T17:54:49.881 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T17:54:50.007 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T17:54:50.111 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:54:50.570 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:54:50.851 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:54:50.851 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:54:50.851 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T17:54:50.851 INFO:teuthology.orchestra.run.smithi066.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:54:50.851 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:54:50.851 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T17:54:51.062 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T17:54:51.656 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 1:34:53 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:54:51.811 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T17:54:51.812 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:54:51.812 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-25T17:54:51.812 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:54:51.812 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T17:54:51.812 INFO:teuthology.orchestra.run.smithi066.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T17:54:51.812 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:54:51.812 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T17:54:51.812 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:54:51.812 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-25T17:54:51.812 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:54:51.812 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 5.4 M 2024-03-25T17:54:51.813 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 25 M 2024-03-25T17:54:51.813 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T17:55:02.514 INFO:teuthology.orchestra.run.smithi193.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 359 kB/s | 44 MB 02:06 2024-03-25T17:55:02.514 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:55:02.515 INFO:teuthology.orchestra.run.smithi193.stdout:Total 359 kB/s | 44 MB 02:06 2024-03-25T17:55:02.515 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T17:55:02.519 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T17:55:02.519 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T17:55:02.818 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T17:55:02.818 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T17:55:07.864 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T17:55:07.964 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:55:09.804 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:55:10.055 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:55:10.055 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:55:10.055 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T17:55:10.055 INFO:teuthology.orchestra.run.smithi193.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:10.055 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:55:10.055 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T17:55:10.244 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install librbd1-debuginfo 2024-03-25T17:55:10.810 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:35:12 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:55:10.961 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T17:55:10.962 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:55:10.962 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T17:55:10.962 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:55:10.963 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T17:55:10.963 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:55:10.963 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:55:10.963 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T17:55:10.963 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:55:10.963 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T17:55:10.963 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:55:10.963 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 52 M 2024-03-25T17:55:10.963 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 308 M 2024-03-25T17:55:10.963 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T17:55:11.994 INFO:teuthology.orchestra.run.smithi066.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 272 kB/s | 5.4 MB 00:20 2024-03-25T17:55:11.995 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:55:11.995 INFO:teuthology.orchestra.run.smithi066.stdout:Total 272 kB/s | 5.4 MB 00:20 2024-03-25T17:55:11.996 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T17:55:12.000 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T17:55:12.000 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T17:55:12.073 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T17:55:12.073 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T17:55:12.773 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T17:55:12.897 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:55:13.433 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:55:13.702 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:55:13.702 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:55:13.702 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T17:55:13.702 INFO:teuthology.orchestra.run.smithi066.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:13.702 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:55:13.703 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T17:55:13.921 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install librados2-debuginfo 2024-03-25T17:55:14.516 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 1:35:16 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:55:14.671 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T17:55:14.672 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:55:14.672 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-25T17:55:14.672 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:55:14.673 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T17:55:14.673 INFO:teuthology.orchestra.run.smithi066.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T17:55:14.673 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:55:14.673 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T17:55:14.673 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:55:14.673 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-25T17:55:14.673 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:55:14.673 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 44 M 2024-03-25T17:55:14.673 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 253 M 2024-03-25T17:55:14.673 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T17:57:37.639 INFO:teuthology.orchestra.run.smithi193.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 362 kB/s | 52 MB 02:26 2024-03-25T17:57:37.640 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:57:37.640 INFO:teuthology.orchestra.run.smithi193.stdout:Total 362 kB/s | 52 MB 02:26 2024-03-25T17:57:37.640 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T17:57:37.643 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T17:57:37.643 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T17:57:37.990 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T17:57:37.990 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T17:57:42.499 INFO:teuthology.orchestra.run.smithi066.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 307 kB/s | 44 MB 02:27 2024-03-25T17:57:42.499 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:57:42.499 INFO:teuthology.orchestra.run.smithi066.stdout:Total 307 kB/s | 44 MB 02:27 2024-03-25T17:57:42.500 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T17:57:42.506 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T17:57:42.506 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T17:57:42.819 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T17:57:42.819 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T17:57:43.970 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T17:57:44.070 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:57:46.240 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:57:46.500 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:57:46.500 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:57:46.500 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T17:57:46.500 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:57:46.501 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:57:46.501 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T17:57:46.700 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T17:57:47.259 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:37:49 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:57:47.409 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T17:57:47.410 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:57:47.410 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T17:57:47.410 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:57:47.410 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T17:57:47.410 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T17:57:47.410 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:57:47.410 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T17:57:47.410 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:57:47.410 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T17:57:47.411 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:57:47.411 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 615 k 2024-03-25T17:57:47.411 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 2.1 M 2024-03-25T17:57:47.411 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T17:57:48.169 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T17:57:48.283 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:57:50.032 INFO:teuthology.orchestra.run.smithi193.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 235 kB/s | 615 kB 00:02 2024-03-25T17:57:50.032 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:57:50.033 INFO:teuthology.orchestra.run.smithi193.stdout:Total 234 kB/s | 615 kB 00:02 2024-03-25T17:57:50.033 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T17:57:50.037 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T17:57:50.037 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T17:57:50.064 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:57:50.078 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T17:57:50.078 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T17:57:50.248 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T17:57:50.327 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:57:50.327 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:57:50.327 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T17:57:50.327 INFO:teuthology.orchestra.run.smithi066.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:57:50.327 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:57:50.327 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T17:57:50.356 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:57:50.528 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install librbd1-debuginfo 2024-03-25T17:57:50.733 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:57:50.978 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:57:50.978 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:57:50.978 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T17:57:50.978 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:57:50.978 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:57:50.978 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T17:57:51.123 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 1:37:53 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:57:51.149 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install python3-rados-debuginfo 2024-03-25T17:57:51.280 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T17:57:51.281 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:57:51.281 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-25T17:57:51.281 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:57:51.281 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T17:57:51.282 INFO:teuthology.orchestra.run.smithi066.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:57:51.282 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:57:51.282 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T17:57:51.282 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:57:51.282 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-25T17:57:51.282 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:57:51.282 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 52 M 2024-03-25T17:57:51.282 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 308 M 2024-03-25T17:57:51.282 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T17:57:51.711 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:37:53 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:57:51.864 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T17:57:51.865 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:57:51.865 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T17:57:51.865 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:57:51.865 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T17:57:51.865 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T17:57:51.865 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:57:51.865 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T17:57:51.865 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:57:51.865 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T17:57:51.865 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:57:51.865 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 771 k 2024-03-25T17:57:51.865 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 2.7 M 2024-03-25T17:57:51.865 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T17:57:54.726 INFO:teuthology.orchestra.run.smithi193.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 269 kB/s | 771 kB 00:02 2024-03-25T17:57:54.727 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:57:54.727 INFO:teuthology.orchestra.run.smithi193.stdout:Total 269 kB/s | 771 kB 00:02 2024-03-25T17:57:54.727 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T17:57:54.731 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T17:57:54.731 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T17:57:54.773 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T17:57:54.774 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T17:57:54.958 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T17:57:55.076 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:57:55.461 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:57:55.703 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:57:55.703 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:57:55.703 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T17:57:55.703 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:57:55.703 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:57:55.703 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T17:57:55.912 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T17:57:56.517 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:37:58 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:57:56.666 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T17:57:56.666 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:57:56.667 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T17:57:56.667 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:57:56.667 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T17:57:56.667 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T17:57:56.667 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:57:56.667 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T17:57:56.667 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:57:56.667 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T17:57:56.667 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:57:56.667 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 1.1 M 2024-03-25T17:57:56.667 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 4.3 M 2024-03-25T17:57:56.667 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T17:58:00.079 INFO:teuthology.orchestra.run.smithi193.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 337 kB/s | 1.1 MB 00:03 2024-03-25T17:58:00.080 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:00.080 INFO:teuthology.orchestra.run.smithi193.stdout:Total 337 kB/s | 1.1 MB 00:03 2024-03-25T17:58:00.080 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T17:58:00.084 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T17:58:00.084 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T17:58:00.127 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T17:58:00.127 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T17:58:00.367 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T17:58:00.478 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:58:00.889 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:58:01.133 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:58:01.133 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:58:01.133 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T17:58:01.133 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:01.133 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:58:01.133 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T17:58:01.334 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T17:58:01.897 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:38:03 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:58:02.046 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T17:58:02.047 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:58:02.047 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T17:58:02.047 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:58:02.048 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T17:58:02.048 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T17:58:02.048 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:58:02.048 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T17:58:02.048 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:58:02.048 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T17:58:02.048 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:58:02.048 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 337 k 2024-03-25T17:58:02.048 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 1.0 M 2024-03-25T17:58:02.048 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T17:58:03.449 INFO:teuthology.orchestra.run.smithi193.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 241 kB/s | 337 kB 00:01 2024-03-25T17:58:03.449 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:03.450 INFO:teuthology.orchestra.run.smithi193.stdout:Total 241 kB/s | 337 kB 00:01 2024-03-25T17:58:03.450 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T17:58:03.454 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T17:58:03.454 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T17:58:03.494 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T17:58:03.494 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T17:58:03.636 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T17:58:03.756 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:58:04.106 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:58:04.350 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:58:04.350 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:58:04.350 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T17:58:04.350 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:04.350 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:58:04.350 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T17:58:04.549 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T17:58:05.108 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:38:07 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:58:05.258 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T17:58:05.258 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:58:05.258 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T17:58:05.258 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:58:05.258 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T17:58:05.258 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T17:58:05.259 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:58:05.259 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T17:58:05.259 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:58:05.259 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T17:58:05.259 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:58:05.259 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 509 k 2024-03-25T17:58:05.259 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 2.5 M 2024-03-25T17:58:05.259 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T17:58:07.093 INFO:teuthology.orchestra.run.smithi193.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 277 kB/s | 509 kB 00:01 2024-03-25T17:58:07.093 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:07.094 INFO:teuthology.orchestra.run.smithi193.stdout:Total 277 kB/s | 509 kB 00:01 2024-03-25T17:58:07.094 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T17:58:07.098 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T17:58:07.098 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T17:58:07.138 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T17:58:07.138 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T17:58:07.299 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T17:58:07.400 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:07.753 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:08.010 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:08.010 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:58:08.010 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T17:58:08.010 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:08.010 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:58:08.010 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T17:58:08.241 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T17:58:08.803 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:38:10 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:58:08.952 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T17:58:08.953 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:58:08.953 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T17:58:08.953 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:58:08.953 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T17:58:08.953 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:58:08.953 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:58:08.954 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T17:58:08.954 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T17:58:08.954 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T17:58:08.954 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T17:58:08.954 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 52 M 2024-03-25T17:58:08.954 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 298 M 2024-03-25T17:58:08.954 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T17:59:28.892 INFO:teuthology.orchestra.run.smithi066.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 544 kB/s | 52 MB 01:37 2024-03-25T17:59:28.892 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:59:28.892 INFO:teuthology.orchestra.run.smithi066.stdout:Total 544 kB/s | 52 MB 01:37 2024-03-25T17:59:28.893 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T17:59:28.896 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T17:59:28.896 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T17:59:29.259 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T17:59:29.260 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T17:59:35.623 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T17:59:35.718 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:59:37.844 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:59:38.122 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:59:38.123 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:59:38.123 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T17:59:38.123 INFO:teuthology.orchestra.run.smithi066.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:59:38.123 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:59:38.123 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T17:59:38.341 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T17:59:38.930 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 1:39:40 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:59:39.085 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T17:59:39.086 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:59:39.086 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-25T17:59:39.086 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:59:39.086 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T17:59:39.086 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T17:59:39.086 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:59:39.086 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T17:59:39.086 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:59:39.086 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-25T17:59:39.087 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:59:39.087 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 615 k 2024-03-25T17:59:39.087 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 2.1 M 2024-03-25T17:59:39.087 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T17:59:42.235 INFO:teuthology.orchestra.run.smithi066.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 195 kB/s | 615 kB 00:03 2024-03-25T17:59:42.236 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:59:42.236 INFO:teuthology.orchestra.run.smithi066.stdout:Total 195 kB/s | 615 kB 00:03 2024-03-25T17:59:42.236 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T17:59:42.240 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T17:59:42.240 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T17:59:42.286 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T17:59:42.287 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T17:59:42.464 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T17:59:42.578 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:59:42.999 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:59:43.278 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:59:43.279 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:59:43.279 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T17:59:43.279 INFO:teuthology.orchestra.run.smithi066.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:59:43.279 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:59:43.279 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T17:59:43.551 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install python3-rados-debuginfo 2024-03-25T17:59:44.146 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 1:39:46 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:59:44.303 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T17:59:44.304 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:59:44.304 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-25T17:59:44.304 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:59:44.304 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T17:59:44.304 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T17:59:44.304 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:59:44.305 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T17:59:44.305 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:59:44.305 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-25T17:59:44.305 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:59:44.305 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 771 k 2024-03-25T17:59:44.305 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 2.7 M 2024-03-25T17:59:44.305 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T17:59:45.355 INFO:teuthology.orchestra.run.smithi066.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 735 kB/s | 771 kB 00:01 2024-03-25T17:59:45.355 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:59:45.355 INFO:teuthology.orchestra.run.smithi066.stdout:Total 734 kB/s | 771 kB 00:01 2024-03-25T17:59:45.356 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T17:59:45.360 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T17:59:45.360 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T17:59:45.405 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T17:59:45.405 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T17:59:45.597 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T17:59:45.697 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:59:46.211 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:59:46.482 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:59:46.483 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:59:46.483 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T17:59:46.483 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:59:46.483 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:59:46.483 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T17:59:46.696 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T17:59:47.288 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 1:39:49 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:59:47.445 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T17:59:47.446 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:59:47.446 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-25T17:59:47.446 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:59:47.447 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T17:59:47.447 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T17:59:47.447 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:59:47.447 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T17:59:47.447 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:59:47.447 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-25T17:59:47.447 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:59:47.447 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 1.1 M 2024-03-25T17:59:47.447 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 4.3 M 2024-03-25T17:59:47.447 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T17:59:49.674 INFO:teuthology.orchestra.run.smithi066.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 517 kB/s | 1.1 MB 00:02 2024-03-25T17:59:49.675 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:59:49.675 INFO:teuthology.orchestra.run.smithi066.stdout:Total 516 kB/s | 1.1 MB 00:02 2024-03-25T17:59:49.676 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T17:59:49.680 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T17:59:49.680 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T17:59:49.728 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T17:59:49.728 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T17:59:49.964 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T17:59:50.077 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:59:50.506 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:59:50.855 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:59:50.855 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:59:50.855 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T17:59:50.855 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:59:50.856 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:59:50.856 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T17:59:51.086 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T17:59:51.681 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 1:39:53 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:59:51.838 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T17:59:51.839 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:59:51.839 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-25T17:59:51.839 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:59:51.839 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T17:59:51.839 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T17:59:51.839 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:59:51.839 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T17:59:51.840 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:59:51.840 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-25T17:59:51.840 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:59:51.840 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 337 k 2024-03-25T17:59:51.840 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 1.0 M 2024-03-25T17:59:51.840 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T17:59:53.427 INFO:teuthology.orchestra.run.smithi066.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 213 kB/s | 337 kB 00:01 2024-03-25T17:59:53.428 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:59:53.428 INFO:teuthology.orchestra.run.smithi066.stdout:Total 212 kB/s | 337 kB 00:01 2024-03-25T17:59:53.428 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T17:59:53.433 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T17:59:53.433 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T17:59:53.477 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T17:59:53.477 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T17:59:53.627 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T17:59:53.846 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:59:54.231 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:59:54.553 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:59:54.554 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:59:54.554 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T17:59:54.554 INFO:teuthology.orchestra.run.smithi066.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:59:54.554 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:59:54.554 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T17:59:54.778 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T17:59:55.371 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 1:39:57 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:59:55.526 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T17:59:55.527 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:59:55.527 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-25T17:59:55.527 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:59:55.527 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T17:59:55.527 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T17:59:55.527 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:59:55.527 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T17:59:55.527 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:59:55.527 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-25T17:59:55.527 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:59:55.528 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 509 k 2024-03-25T17:59:55.528 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 2.5 M 2024-03-25T17:59:55.528 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T17:59:57.663 INFO:teuthology.orchestra.run.smithi066.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 238 kB/s | 509 kB 00:02 2024-03-25T17:59:57.664 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:59:57.664 INFO:teuthology.orchestra.run.smithi066.stdout:Total 238 kB/s | 509 kB 00:02 2024-03-25T17:59:57.664 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T17:59:57.669 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T17:59:57.669 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T17:59:57.712 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T17:59:57.713 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T17:59:57.885 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T17:59:58.000 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:59:58.404 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:59:58.669 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:59:58.669 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:59:58.670 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T17:59:58.670 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:59:58.670 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:59:58.670 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T17:59:58.879 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T17:59:59.477 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 1:40:01 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T17:59:59.634 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T17:59:59.634 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:59:59.635 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-25T17:59:59.635 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:59:59.635 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T17:59:59.635 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:59:59.635 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:59:59.635 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T17:59:59.635 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T17:59:59.635 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-25T17:59:59.635 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T17:59:59.635 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 52 M 2024-03-25T17:59:59.635 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 298 M 2024-03-25T17:59:59.635 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T18:00:39.666 INFO:teuthology.orchestra.run.smithi193.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 350 kB/s | 52 MB 02:30 2024-03-25T18:00:39.667 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:00:39.667 INFO:teuthology.orchestra.run.smithi193.stdout:Total 350 kB/s | 52 MB 02:30 2024-03-25T18:00:39.667 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T18:00:39.671 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T18:00:39.671 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T18:00:40.007 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T18:00:40.007 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T18:00:45.885 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T18:00:45.989 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:00:48.021 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:00:48.292 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:00:48.292 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:00:48.292 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T18:00:48.292 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:48.292 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:00:48.292 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:00:48.498 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:00:49.059 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 1:40:51 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T18:00:49.209 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-25T18:00:49.210 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:00:49.210 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repo Size 2024-03-25T18:00:49.210 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:00:49.210 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-25T18:00:49.210 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:00:49.210 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:00:49.210 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-25T18:00:49.210 INFO:teuthology.orchestra.run.smithi193.stdout:================================================================================ 2024-03-25T18:00:49.210 INFO:teuthology.orchestra.run.smithi193.stdout:Install 1 Package 2024-03-25T18:00:49.210 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:00:49.210 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 1.3 M 2024-03-25T18:00:49.211 INFO:teuthology.orchestra.run.smithi193.stdout:Installed size: 5.5 M 2024-03-25T18:00:49.211 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-25T18:00:52.501 INFO:teuthology.orchestra.run.smithi193.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 390 kB/s | 1.3 MB 00:03 2024-03-25T18:00:52.501 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:00:52.502 INFO:teuthology.orchestra.run.smithi193.stdout:Total 389 kB/s | 1.3 MB 00:03 2024-03-25T18:00:52.502 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-25T18:00:52.506 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-25T18:00:52.506 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-25T18:00:52.550 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-25T18:00:52.550 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-25T18:00:52.797 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-25T18:00:52.907 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:00:53.300 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:00:53.548 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:00:53.549 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:00:53.549 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-25T18:00:53.549 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:53.549 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:00:53.549 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-25T18:00:53.741 DEBUG:teuthology.parallel:result is None 2024-03-25T18:02:36.135 INFO:teuthology.orchestra.run.smithi066.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 337 kB/s | 52 MB 02:36 2024-03-25T18:02:36.136 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:02:36.136 INFO:teuthology.orchestra.run.smithi066.stdout:Total 337 kB/s | 52 MB 02:36 2024-03-25T18:02:36.136 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T18:02:36.139 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T18:02:36.139 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T18:02:36.491 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T18:02:36.491 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T18:02:42.766 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T18:02:42.869 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:02:44.949 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:02:45.218 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:02:45.219 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:45.219 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T18:02:45.219 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:45.219 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:45.219 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T18:02:45.423 DEBUG:teuthology.orchestra.run.smithi066:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:02:46.024 INFO:teuthology.orchestra.run.smithi066.stdout:Last metadata expiration check: 1:42:48 ago on Mon 25 Mar 2024 04:19:58 PM UTC. 2024-03-25T18:02:46.179 INFO:teuthology.orchestra.run.smithi066.stdout:Dependencies resolved. 2024-03-25T18:02:46.180 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T18:02:46.180 INFO:teuthology.orchestra.run.smithi066.stdout: Package Arch Version Repo Size 2024-03-25T18:02:46.180 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T18:02:46.180 INFO:teuthology.orchestra.run.smithi066.stdout:Installing: 2024-03-25T18:02:46.180 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:02:46.180 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:46.180 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction Summary 2024-03-25T18:02:46.181 INFO:teuthology.orchestra.run.smithi066.stdout:================================================================================ 2024-03-25T18:02:46.181 INFO:teuthology.orchestra.run.smithi066.stdout:Install 1 Package 2024-03-25T18:02:46.181 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:46.181 INFO:teuthology.orchestra.run.smithi066.stdout:Total download size: 1.3 M 2024-03-25T18:02:46.181 INFO:teuthology.orchestra.run.smithi066.stdout:Installed size: 5.5 M 2024-03-25T18:02:46.181 INFO:teuthology.orchestra.run.smithi066.stdout:Downloading Packages: 2024-03-25T18:02:51.378 INFO:teuthology.orchestra.run.smithi066.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 247 kB/s | 1.3 MB 00:05 2024-03-25T18:02:51.378 INFO:teuthology.orchestra.run.smithi066.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:02:51.379 INFO:teuthology.orchestra.run.smithi066.stdout:Total 247 kB/s | 1.3 MB 00:05 2024-03-25T18:02:51.379 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction check 2024-03-25T18:02:51.383 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction check succeeded. 2024-03-25T18:02:51.383 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction test 2024-03-25T18:02:51.433 INFO:teuthology.orchestra.run.smithi066.stdout:Transaction test succeeded. 2024-03-25T18:02:51.433 INFO:teuthology.orchestra.run.smithi066.stdout:Running transaction 2024-03-25T18:02:51.685 INFO:teuthology.orchestra.run.smithi066.stdout: Preparing : 1/1 2024-03-25T18:02:51.788 INFO:teuthology.orchestra.run.smithi066.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:02:52.212 INFO:teuthology.orchestra.run.smithi066.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:02:52.499 INFO:teuthology.orchestra.run.smithi066.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:02:52.499 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:52.499 INFO:teuthology.orchestra.run.smithi066.stdout:Installed: 2024-03-25T18:02:52.499 INFO:teuthology.orchestra.run.smithi066.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:52.499 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:52.499 INFO:teuthology.orchestra.run.smithi066.stdout:Complete! 2024-03-25T18:02:52.700 DEBUG:teuthology.parallel:result is None 2024-03-25T18:02:52.701 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:02:52.844 DEBUG:teuthology.orchestra.run.smithi066:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:02:52.866 INFO:teuthology.orchestra.run.smithi066.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:02:52.866 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:02:52.866 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:02:52.868 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:02:53.018 DEBUG:teuthology.orchestra.run.smithi193:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:02:53.043 INFO:teuthology.orchestra.run.smithi193.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:02:53.043 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:02:53.043 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:02:53.045 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T18:02:53.045 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-25T18:02:53.045 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:02:53.076 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T18:02:53.076 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:02:53.115 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T18:02:53.115 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-25T18:02:53.115 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:02:53.148 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:02:53.219 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T18:02:53.219 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:02:53.248 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:02:53.317 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T18:02:53.317 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-25T18:02:53.318 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:02:53.348 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:02:53.418 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T18:02:53.418 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:02:53.446 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:02:53.514 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T18:02:53.515 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-25T18:02:53.515 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:02:53.545 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:02:53.616 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T18:02:53.616 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:02:53.644 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:02:53.712 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T18:02:53.797 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T18:02:53.797 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 777 /var/log/ceph 2024-03-25T18:02:53.800 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 777 /var/log/ceph 2024-03-25T18:02:53.828 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T18:02:53.828 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:02:53.870 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:02:53.898 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T18:02:53.898 DEBUG:teuthology.orchestra.run.smithi066:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:02:53.941 DEBUG:teuthology.orchestra.run.smithi193:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:02:54.016 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T18:02:54.016 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T18:02:54.017 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621249', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621249', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi066.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHiFRiyARlFOX6EzyMAmT7g0HngrubTPU4jsSfsyWrlEflazx4IQRo9ara2llBy4NEOGryTG/sLr+mPufh99/n8=', 'smithi193.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDCcALhUKDoQ73A9zVvrTXTBClHg7qAmouPpjds8k2OzS4rF5FKG4D5lDjy5mM8V+0RU7Lck9wnVivw9YWcRFI8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T18:02:54.017 DEBUG:teuthology.orchestra.run.smithi066:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:02:54.020 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:02:54.034 DEBUG:teuthology.orchestra.run.smithi066:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:02:54.075 DEBUG:teuthology.orchestra.run.smithi193:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:02:54.103 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-25T18:02:54.103 DEBUG:teuthology.orchestra.run.smithi066:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:02:54.159 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:02:54.160 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_1 2024-03-25T18:02:54.215 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:02:54.215 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:02:54.215 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-25T18:02:54.215 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:02:54.215 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:02:54.215 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-03-25 16:23:33.636703868 +0000 2024-03-25T18:02:54.215 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-03-25 16:18:53.742038658 +0000 2024-03-25T18:02:54.215 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-03-25 16:18:53.742038658 +0000 2024-03-25T18:02:54.215 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-03-25 16:18:53.742038658 +0000 2024-03-25T18:02:54.215 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:02:54.284 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-03-25T18:02:54.284 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-03-25T18:02:54.284 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000256492 s, 2.0 MB/s 2024-03-25T18:02:54.285 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:02:54.342 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_2 2024-03-25T18:02:54.397 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:02:54.397 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:02:54.397 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-25T18:02:54.397 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:02:54.397 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:02:54.397 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-03-25 16:23:33.637703860 +0000 2024-03-25T18:02:54.397 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-03-25 16:18:54.097035744 +0000 2024-03-25T18:02:54.397 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-03-25 16:18:54.097035744 +0000 2024-03-25T18:02:54.397 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-03-25 16:18:54.097035744 +0000 2024-03-25T18:02:54.398 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:02:54.466 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-03-25T18:02:54.466 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-03-25T18:02:54.466 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000172108 s, 3.0 MB/s 2024-03-25T18:02:54.467 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:02:54.522 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_3 2024-03-25T18:02:54.578 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:02:54.578 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:02:54.578 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-25T18:02:54.578 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:02:54.578 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:02:54.578 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-03-25 16:23:33.637703860 +0000 2024-03-25T18:02:54.578 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-03-25 16:18:54.447032871 +0000 2024-03-25T18:02:54.578 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-03-25 16:18:54.447032871 +0000 2024-03-25T18:02:54.578 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-03-25 16:18:54.447032871 +0000 2024-03-25T18:02:54.579 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:02:54.647 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-03-25T18:02:54.647 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-03-25T18:02:54.647 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.000198236 s, 2.6 MB/s 2024-03-25T18:02:54.648 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:02:54.703 DEBUG:teuthology.orchestra.run.smithi066:> stat /dev/vg_nvme/lv_4 2024-03-25T18:02:54.759 INFO:teuthology.orchestra.run.smithi066.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:02:54.759 INFO:teuthology.orchestra.run.smithi066.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:02:54.759 INFO:teuthology.orchestra.run.smithi066.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-25T18:02:54.759 INFO:teuthology.orchestra.run.smithi066.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:02:54.759 INFO:teuthology.orchestra.run.smithi066.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:02:54.759 INFO:teuthology.orchestra.run.smithi066.stdout:Access: 2024-03-25 16:23:33.637703860 +0000 2024-03-25T18:02:54.759 INFO:teuthology.orchestra.run.smithi066.stdout:Modify: 2024-03-25 16:18:54.808029908 +0000 2024-03-25T18:02:54.759 INFO:teuthology.orchestra.run.smithi066.stdout:Change: 2024-03-25 16:18:54.808029908 +0000 2024-03-25T18:02:54.759 INFO:teuthology.orchestra.run.smithi066.stdout: Birth: 2024-03-25 16:18:54.808029908 +0000 2024-03-25T18:02:54.759 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:02:54.828 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records in 2024-03-25T18:02:54.828 INFO:teuthology.orchestra.run.smithi066.stderr:1+0 records out 2024-03-25T18:02:54.828 INFO:teuthology.orchestra.run.smithi066.stderr:512 bytes copied, 0.00020122 s, 2.5 MB/s 2024-03-25T18:02:54.829 DEBUG:teuthology.orchestra.run.smithi066:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:02:54.884 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:02:54.884 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T18:02:54.884 DEBUG:teuthology.orchestra.run.smithi193:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:02:54.902 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:02:54.902 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_1 2024-03-25T18:02:54.959 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:02:54.959 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:02:54.959 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-25T18:02:54.959 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:02:54.959 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:02:54.959 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-03-25 16:23:14.208295052 +0000 2024-03-25T18:02:54.959 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-03-25 16:18:52.730251024 +0000 2024-03-25T18:02:54.959 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-03-25 16:18:52.730251024 +0000 2024-03-25T18:02:54.960 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-03-25 16:18:52.730251024 +0000 2024-03-25T18:02:54.960 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:02:55.029 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-03-25T18:02:55.029 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-03-25T18:02:55.029 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000223182 s, 2.3 MB/s 2024-03-25T18:02:55.030 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:02:55.088 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_2 2024-03-25T18:02:55.143 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:02:55.144 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:02:55.144 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-25T18:02:55.144 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:02:55.144 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:02:55.144 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-03-25 16:23:14.208295052 +0000 2024-03-25T18:02:55.144 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-03-25 16:18:53.078244999 +0000 2024-03-25T18:02:55.144 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-03-25 16:18:53.078244999 +0000 2024-03-25T18:02:55.144 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-03-25 16:18:53.078244999 +0000 2024-03-25T18:02:55.144 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:02:55.210 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-03-25T18:02:55.210 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-03-25T18:02:55.211 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000189855 s, 2.7 MB/s 2024-03-25T18:02:55.212 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:02:55.268 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_3 2024-03-25T18:02:55.324 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:02:55.324 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:02:55.324 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-25T18:02:55.324 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:02:55.324 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:02:55.324 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-03-25 16:23:14.209295035 +0000 2024-03-25T18:02:55.325 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-03-25 16:18:53.424239010 +0000 2024-03-25T18:02:55.325 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-03-25 16:18:53.424239010 +0000 2024-03-25T18:02:55.325 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-03-25 16:18:53.424239010 +0000 2024-03-25T18:02:55.325 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:02:55.391 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-03-25T18:02:55.391 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-03-25T18:02:55.391 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000230148 s, 2.2 MB/s 2024-03-25T18:02:55.393 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:02:55.449 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_4 2024-03-25T18:02:55.505 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:02:55.505 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:02:55.505 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-25T18:02:55.505 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:02:55.505 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:02:55.506 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-03-25 16:23:14.209295035 +0000 2024-03-25T18:02:55.506 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-03-25 16:18:53.767233072 +0000 2024-03-25T18:02:55.506 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-03-25 16:18:53.767233072 +0000 2024-03-25T18:02:55.506 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-03-25 16:18:53.767233072 +0000 2024-03-25T18:02:55.506 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:02:55.571 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-03-25T18:02:55.571 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-03-25T18:02:55.572 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000195183 s, 2.6 MB/s 2024-03-25T18:02:55.573 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:02:55.630 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:02:55.630 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi066.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi193.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:02:55.630 INFO:tasks.ceph:Generating config... 2024-03-25T18:02:55.632 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T18:02:55.632 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T18:02:55.632 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T18:02:55.632 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T18:02:55.632 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T18:02:55.632 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T18:02:55.633 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T18:02:55.633 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T18:02:55.633 INFO:tasks.ceph:[global] ms type = async 2024-03-25T18:02:55.633 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T18:02:55.633 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T18:02:55.633 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T18:02:55.633 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T18:02:55.633 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T18:02:55.634 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T18:02:55.634 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T18:02:55.634 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T18:02:55.634 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T18:02:55.634 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T18:02:55.634 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T18:02:55.634 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T18:02:55.635 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T18:02:55.635 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T18:02:55.635 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T18:02:55.635 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T18:02:55.635 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T18:02:55.635 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T18:02:55.635 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T18:02:55.635 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T18:02:55.635 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T18:02:55.636 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T18:02:55.636 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T18:02:55.636 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T18:02:55.636 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T18:02:55.636 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T18:02:55.636 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T18:02:55.636 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T18:02:55.636 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T18:02:55.637 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T18:02:55.637 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T18:02:55.637 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T18:02:55.637 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T18:02:55.637 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T18:02:55.637 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T18:02:55.637 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T18:02:55.637 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T18:02:55.638 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T18:02:55.638 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T18:02:55.638 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T18:02:55.638 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T18:02:55.638 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T18:02:55.638 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T18:02:55.638 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T18:02:55.639 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T18:02:55.639 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T18:02:55.700 INFO:teuthology.orchestra.run.smithi066.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T18:02:55.703 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T18:02:55.754 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:02:55.787 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.66:6789]'), ('mon.c', '[v1:172.21.15.66:6790]'), ('mon.b', '[v1:172.21.15.193:6789]')] 2024-03-25T18:02:55.787 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.66:6789],[v1:172.21.15.66:6790],[v1:172.21.15.193:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T18:02:55.789 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-25T18:02:55.789 DEBUG:teuthology.orchestra.run.smithi066:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:02:55.843 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.66:6789]' --addv c '[v1:172.21.15.66:6790]' --addv b '[v1:172.21.15.193:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:02:55.928 INFO:teuthology.orchestra.run.smithi066.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:02:55.928 INFO:teuthology.orchestra.run.smithi066.stdout:monmaptool: generated fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:02:55.928 INFO:teuthology.orchestra.run.smithi066.stdout:setting min_mon_release = quincy 2024-03-25T18:02:55.928 INFO:teuthology.orchestra.run.smithi066.stdout:epoch 0 2024-03-25T18:02:55.929 INFO:teuthology.orchestra.run.smithi066.stdout:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:02:55.929 INFO:teuthology.orchestra.run.smithi066.stdout:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:02:55.929 INFO:teuthology.orchestra.run.smithi066.stdout:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:02:55.929 INFO:teuthology.orchestra.run.smithi066.stdout:min_mon_release 17 (quincy) 2024-03-25T18:02:55.929 INFO:teuthology.orchestra.run.smithi066.stdout:election_strategy: 3 2024-03-25T18:02:55.929 INFO:teuthology.orchestra.run.smithi066.stdout:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:02:55.929 INFO:teuthology.orchestra.run.smithi066.stdout:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:02:55.929 INFO:teuthology.orchestra.run.smithi066.stdout:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:02:55.929 INFO:teuthology.orchestra.run.smithi066.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T18:02:55.930 DEBUG:teuthology.orchestra.run.smithi066:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:02:55.945 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 8da823bc-9cd7-40a4-b5f1-07036a791c17... 2024-03-25T18:02:55.946 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:02:55.987 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:02:56.038 INFO:teuthology.orchestra.run.smithi193.stdout:[global] 2024-03-25T18:02:56.039 INFO:teuthology.orchestra.run.smithi193.stdout: chdir = "" 2024-03-25T18:02:56.039 INFO:teuthology.orchestra.run.smithi193.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:02:56.039 INFO:teuthology.orchestra.run.smithi193.stdout: auth supported = cephx 2024-03-25T18:02:56.039 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:02:56.039 INFO:teuthology.orchestra.run.smithi193.stdout: filestore xattr use omap = true 2024-03-25T18:02:56.039 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:02:56.039 INFO:teuthology.orchestra.run.smithi193.stdout: mon clock drift allowed = 1.000 2024-03-25T18:02:56.039 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:02:56.039 INFO:teuthology.orchestra.run.smithi193.stdout: osd crush chooseleaf type = 0 2024-03-25T18:02:56.039 INFO:teuthology.orchestra.run.smithi193.stdout: auth debug = true 2024-03-25T18:02:56.039 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:02:56.039 INFO:teuthology.orchestra.run.smithi193.stdout: ms die on old message = true 2024-03-25T18:02:56.039 INFO:teuthology.orchestra.run.smithi193.stdout: ms die on bug = true 2024-03-25T18:02:56.039 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:02:56.040 INFO:teuthology.orchestra.run.smithi193.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:02:56.040 INFO:teuthology.orchestra.run.smithi193.stdout: mon pg warn max object skew = 0 2024-03-25T18:02:56.040 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:02:56.040 INFO:teuthology.orchestra.run.smithi193.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:02:56.040 INFO:teuthology.orchestra.run.smithi193.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:02:56.040 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:02:56.040 INFO:teuthology.orchestra.run.smithi193.stdout: osd pool default size = 2 2024-03-25T18:02:56.040 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:02:56.040 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd allow primary affinity = true 2024-03-25T18:02:56.040 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd allow pg remap = true 2024-03-25T18:02:56.040 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:02:56.040 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:02:56.040 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on no sortbitwise = false 2024-03-25T18:02:56.040 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:02:56.040 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on too few osds = false 2024-03-25T18:02:56.040 INFO:teuthology.orchestra.run.smithi193.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:02:56.041 INFO:teuthology.orchestra.run.smithi193.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:02:56.041 INFO:teuthology.orchestra.run.smithi193.stdout: mon_allow_pool_size_one = true 2024-03-25T18:02:56.041 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:02:56.041 INFO:teuthology.orchestra.run.smithi193.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:02:56.041 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:02:56.041 INFO:teuthology.orchestra.run.smithi193.stdout: osd default data pool replay window = 5 2024-03-25T18:02:56.041 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:02:56.041 INFO:teuthology.orchestra.run.smithi193.stdout: mon allow pool delete = true 2024-03-25T18:02:56.041 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:02:56.041 INFO:teuthology.orchestra.run.smithi193.stdout: mon cluster log file level = debug 2024-03-25T18:02:56.041 INFO:teuthology.orchestra.run.smithi193.stdout: debug asserts on shutdown = true 2024-03-25T18:02:56.041 INFO:teuthology.orchestra.run.smithi193.stdout: mon health detail to clog = false 2024-03-25T18:02:56.041 INFO:teuthology.orchestra.run.smithi193.stdout: mon host = "[v1:172.21.15.66:6789],[v1:172.21.15.66:6790],[v1:172.21.15.193:6789]" 2024-03-25T18:02:56.042 INFO:teuthology.orchestra.run.smithi193.stdout: mon client directed command retry = 5 2024-03-25T18:02:56.042 INFO:teuthology.orchestra.run.smithi193.stdout: mon election default strategy = 3 2024-03-25T18:02:56.042 INFO:teuthology.orchestra.run.smithi193.stdout: ms bind msgr2 = False 2024-03-25T18:02:56.042 INFO:teuthology.orchestra.run.smithi193.stdout: ms inject socket failures = 5000 2024-03-25T18:02:56.042 INFO:teuthology.orchestra.run.smithi193.stdout: ms type = async 2024-03-25T18:02:56.042 INFO:teuthology.orchestra.run.smithi193.stdout: osd heartbeat grace = 80 2024-03-25T18:02:56.042 INFO:teuthology.orchestra.run.smithi193.stdout: fsid = 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:02:56.042 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:02:56.042 INFO:teuthology.orchestra.run.smithi193.stdout:[osd] 2024-03-25T18:02:56.042 INFO:teuthology.orchestra.run.smithi193.stdout: osd journal size = 100 2024-03-25T18:02:56.042 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:02:56.042 INFO:teuthology.orchestra.run.smithi193.stdout: osd scrub load threshold = 5.0 2024-03-25T18:02:56.042 INFO:teuthology.orchestra.run.smithi193.stdout: osd scrub max interval = 600 2024-03-25T18:02:56.042 INFO:teuthology.orchestra.run.smithi193.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:02:56.042 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:02:56.043 INFO:teuthology.orchestra.run.smithi193.stdout: osd recover clone overlap = true 2024-03-25T18:02:56.043 INFO:teuthology.orchestra.run.smithi193.stdout: osd recovery max chunk = 1048576 2024-03-25T18:02:56.043 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:02:56.043 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug shutdown = true 2024-03-25T18:02:56.043 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug op order = true 2024-03-25T18:02:56.043 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug verify stray on activate = true 2024-03-25T18:02:56.043 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug trim objects = true 2024-03-25T18:02:56.043 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:02:56.043 INFO:teuthology.orchestra.run.smithi193.stdout: osd open classes on start = true 2024-03-25T18:02:56.043 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug pg log writeout = true 2024-03-25T18:02:56.043 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:02:56.043 INFO:teuthology.orchestra.run.smithi193.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:02:56.043 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:02:56.043 INFO:teuthology.orchestra.run.smithi193.stdout: osd map max advance = 10 2024-03-25T18:02:56.043 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:02:56.044 INFO:teuthology.orchestra.run.smithi193.stdout: journal zero on create = true 2024-03-25T18:02:56.044 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:02:56.044 INFO:teuthology.orchestra.run.smithi193.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:02:56.044 INFO:teuthology.orchestra.run.smithi193.stdout: filestore apply finisher threads = 3 2024-03-25T18:02:56.044 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:02:56.044 INFO:teuthology.orchestra.run.smithi193.stdout: bdev debug aio = true 2024-03-25T18:02:56.044 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug misdirected ops = true 2024-03-25T18:02:56.044 INFO:teuthology.orchestra.run.smithi193.stdout: bdev async discard = True 2024-03-25T18:02:56.044 INFO:teuthology.orchestra.run.smithi193.stdout: bdev enable discard = True 2024-03-25T18:02:56.044 INFO:teuthology.orchestra.run.smithi193.stdout: bluestore allocator = stupid 2024-03-25T18:02:56.044 INFO:teuthology.orchestra.run.smithi193.stdout: bluestore block size = 96636764160 2024-03-25T18:02:56.044 INFO:teuthology.orchestra.run.smithi193.stdout: bluestore fsck on mount = True 2024-03-25T18:02:56.044 INFO:teuthology.orchestra.run.smithi193.stdout: bluestore zero block detection = True 2024-03-25T18:02:56.044 INFO:teuthology.orchestra.run.smithi193.stdout: debug bluefs = 1 2024-03-25T18:02:56.044 INFO:teuthology.orchestra.run.smithi193.stdout: debug bluestore = 1 2024-03-25T18:02:56.044 INFO:teuthology.orchestra.run.smithi193.stdout: debug monc = 20 2024-03-25T18:02:56.045 INFO:teuthology.orchestra.run.smithi193.stdout: debug ms = 1 2024-03-25T18:02:56.045 INFO:teuthology.orchestra.run.smithi193.stdout: debug osd = 20 2024-03-25T18:02:56.045 INFO:teuthology.orchestra.run.smithi193.stdout: debug rocksdb = 10 2024-03-25T18:02:56.045 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:02:56.045 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd full ratio = 0.9 2024-03-25T18:02:56.045 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:02:56.045 INFO:teuthology.orchestra.run.smithi193.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:02:56.045 INFO:teuthology.orchestra.run.smithi193.stdout: osd class default list = * 2024-03-25T18:02:56.045 INFO:teuthology.orchestra.run.smithi193.stdout: osd class load list = * 2024-03-25T18:02:56.045 INFO:teuthology.orchestra.run.smithi193.stdout: osd client watch timeout = 120 2024-03-25T18:02:56.045 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug verify cached snaps = True 2024-03-25T18:02:56.045 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug verify missing on start = True 2024-03-25T18:02:56.045 INFO:teuthology.orchestra.run.smithi193.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:02:56.045 INFO:teuthology.orchestra.run.smithi193.stdout: osd fast shutdown = False 2024-03-25T18:02:56.045 INFO:teuthology.orchestra.run.smithi193.stdout: osd max markdown count = 1000 2024-03-25T18:02:56.045 INFO:teuthology.orchestra.run.smithi193.stdout: osd mclock override recovery settings = True 2024-03-25T18:02:56.046 INFO:teuthology.orchestra.run.smithi193.stdout: osd objectstore = bluestore 2024-03-25T18:02:56.046 INFO:teuthology.orchestra.run.smithi193.stdout: osd op queue = debug_random 2024-03-25T18:02:56.046 INFO:teuthology.orchestra.run.smithi193.stdout: osd op queue cut off = debug_random 2024-03-25T18:02:56.046 INFO:teuthology.orchestra.run.smithi193.stdout: osd shutdown pgref assert = True 2024-03-25T18:02:56.046 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:02:56.046 INFO:teuthology.orchestra.run.smithi193.stdout:[mgr] 2024-03-25T18:02:56.046 INFO:teuthology.orchestra.run.smithi193.stdout: debug ms = 1 2024-03-25T18:02:56.046 INFO:teuthology.orchestra.run.smithi193.stdout: debug mgr = 20 2024-03-25T18:02:56.046 INFO:teuthology.orchestra.run.smithi193.stdout: debug mon = 20 2024-03-25T18:02:56.046 INFO:teuthology.orchestra.run.smithi193.stdout: debug auth = 20 2024-03-25T18:02:56.046 INFO:teuthology.orchestra.run.smithi193.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:02:56.046 INFO:teuthology.orchestra.run.smithi193.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:02:56.046 INFO:teuthology.orchestra.run.smithi193.stdout: mgr/telemetry/nag = false 2024-03-25T18:02:56.046 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:02:56.046 INFO:teuthology.orchestra.run.smithi193.stdout:[mon] 2024-03-25T18:02:56.047 INFO:teuthology.orchestra.run.smithi193.stdout: debug ms = 1 2024-03-25T18:02:56.047 INFO:teuthology.orchestra.run.smithi193.stdout: debug mon = 20 2024-03-25T18:02:56.047 INFO:teuthology.orchestra.run.smithi193.stdout: debug paxos = 20 2024-03-25T18:02:56.047 INFO:teuthology.orchestra.run.smithi193.stdout: debug auth = 20 2024-03-25T18:02:56.047 INFO:teuthology.orchestra.run.smithi193.stdout: mon data avail warn = 5 2024-03-25T18:02:56.047 INFO:teuthology.orchestra.run.smithi193.stdout: mon mgr mkfs grace = 240 2024-03-25T18:02:56.047 INFO:teuthology.orchestra.run.smithi193.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:02:56.047 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd reporter subtree level = osd 2024-03-25T18:02:56.047 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd prime pg temp = true 2024-03-25T18:02:56.047 INFO:teuthology.orchestra.run.smithi193.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:02:56.047 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:02:56.047 INFO:teuthology.orchestra.run.smithi193.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:02:56.047 INFO:teuthology.orchestra.run.smithi193.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:02:56.047 INFO:teuthology.orchestra.run.smithi193.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:02:56.047 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:02:56.047 INFO:teuthology.orchestra.run.smithi193.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:02:56.047 INFO:teuthology.orchestra.run.smithi193.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:02:56.048 INFO:teuthology.orchestra.run.smithi193.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:02:56.048 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:02:56.048 INFO:teuthology.orchestra.run.smithi193.stdout: # 1m isn't quite enough 2024-03-25T18:02:56.048 INFO:teuthology.orchestra.run.smithi193.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:02:56.048 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:02:56.048 INFO:teuthology.orchestra.run.smithi193.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:02:56.048 INFO:teuthology.orchestra.run.smithi193.stdout: mon min osdmap epochs = 50 2024-03-25T18:02:56.048 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd crush smoke test = False 2024-03-25T18:02:56.048 INFO:teuthology.orchestra.run.smithi193.stdout: mon osdmap full prune interval = 2 2024-03-25T18:02:56.048 INFO:teuthology.orchestra.run.smithi193.stdout: mon osdmap full prune min = 15 2024-03-25T18:02:56.048 INFO:teuthology.orchestra.run.smithi193.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:02:56.048 INFO:teuthology.orchestra.run.smithi193.stdout: mon scrub interval = 300 2024-03-25T18:02:56.048 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on pool no app = False 2024-03-25T18:02:56.048 INFO:teuthology.orchestra.run.smithi193.stdout: paxos service trim min = 10 2024-03-25T18:02:56.048 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-25T18:02:56.048 INFO:teuthology.orchestra.run.smithi193.stdout:[client] 2024-03-25T18:02:56.049 INFO:teuthology.orchestra.run.smithi193.stdout: rgw cache enabled = true 2024-03-25T18:02:56.049 INFO:teuthology.orchestra.run.smithi193.stdout: rgw enable ops log = true 2024-03-25T18:02:56.049 INFO:teuthology.orchestra.run.smithi193.stdout: rgw enable usage log = true 2024-03-25T18:02:56.049 INFO:teuthology.orchestra.run.smithi193.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:02:56.049 INFO:teuthology.orchestra.run.smithi193.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:02:56.049 INFO:teuthology.orchestra.run.smithi193.stdout: debug monc = 20 2024-03-25T18:02:56.049 INFO:teuthology.orchestra.run.smithi193.stdout: debug ms = 1 2024-03-25T18:02:56.049 INFO:teuthology.orchestra.run.smithi193.stdout: debug objecter = 20 2024-03-25T18:02:56.049 INFO:teuthology.orchestra.run.smithi193.stdout: debug rados = 20 2024-03-25T18:02:56.049 INFO:teuthology.orchestra.run.smithi193.stdout:[mon.a] 2024-03-25T18:02:56.049 INFO:teuthology.orchestra.run.smithi193.stdout:[mon.c] 2024-03-25T18:02:56.049 INFO:teuthology.orchestra.run.smithi193.stdout:[mon.b] 2024-03-25T18:02:56.050 INFO:teuthology.orchestra.run.smithi066.stdout:[global] 2024-03-25T18:02:56.050 INFO:teuthology.orchestra.run.smithi066.stdout: chdir = "" 2024-03-25T18:02:56.050 INFO:teuthology.orchestra.run.smithi066.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:02:56.050 INFO:teuthology.orchestra.run.smithi066.stdout: auth supported = cephx 2024-03-25T18:02:56.050 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:56.050 INFO:teuthology.orchestra.run.smithi066.stdout: filestore xattr use omap = true 2024-03-25T18:02:56.050 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:56.050 INFO:teuthology.orchestra.run.smithi066.stdout: mon clock drift allowed = 1.000 2024-03-25T18:02:56.050 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:56.050 INFO:teuthology.orchestra.run.smithi066.stdout: osd crush chooseleaf type = 0 2024-03-25T18:02:56.050 INFO:teuthology.orchestra.run.smithi066.stdout: auth debug = true 2024-03-25T18:02:56.050 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:56.050 INFO:teuthology.orchestra.run.smithi066.stdout: ms die on old message = true 2024-03-25T18:02:56.050 INFO:teuthology.orchestra.run.smithi066.stdout: ms die on bug = true 2024-03-25T18:02:56.051 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:56.051 INFO:teuthology.orchestra.run.smithi066.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:02:56.051 INFO:teuthology.orchestra.run.smithi066.stdout: mon pg warn max object skew = 0 2024-03-25T18:02:56.051 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:56.051 INFO:teuthology.orchestra.run.smithi066.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:02:56.051 INFO:teuthology.orchestra.run.smithi066.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:02:56.051 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:56.051 INFO:teuthology.orchestra.run.smithi066.stdout: osd pool default size = 2 2024-03-25T18:02:56.051 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:56.051 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd allow primary affinity = true 2024-03-25T18:02:56.051 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd allow pg remap = true 2024-03-25T18:02:56.051 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:02:56.051 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:02:56.051 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on no sortbitwise = false 2024-03-25T18:02:56.051 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:02:56.051 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on too few osds = false 2024-03-25T18:02:56.052 INFO:teuthology.orchestra.run.smithi066.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:02:56.052 INFO:teuthology.orchestra.run.smithi066.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:02:56.052 INFO:teuthology.orchestra.run.smithi066.stdout: mon_allow_pool_size_one = true 2024-03-25T18:02:56.052 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:56.052 INFO:teuthology.orchestra.run.smithi066.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:02:56.052 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:56.052 INFO:teuthology.orchestra.run.smithi066.stdout: osd default data pool replay window = 5 2024-03-25T18:02:56.052 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:56.052 INFO:teuthology.orchestra.run.smithi066.stdout: mon allow pool delete = true 2024-03-25T18:02:56.052 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:56.052 INFO:teuthology.orchestra.run.smithi066.stdout: mon cluster log file level = debug 2024-03-25T18:02:56.052 INFO:teuthology.orchestra.run.smithi066.stdout: debug asserts on shutdown = true 2024-03-25T18:02:56.052 INFO:teuthology.orchestra.run.smithi066.stdout: mon health detail to clog = false 2024-03-25T18:02:56.052 INFO:teuthology.orchestra.run.smithi066.stdout: mon host = "[v1:172.21.15.66:6789],[v1:172.21.15.66:6790],[v1:172.21.15.193:6789]" 2024-03-25T18:02:56.052 INFO:teuthology.orchestra.run.smithi066.stdout: mon client directed command retry = 5 2024-03-25T18:02:56.052 INFO:teuthology.orchestra.run.smithi066.stdout: mon election default strategy = 3 2024-03-25T18:02:56.053 INFO:teuthology.orchestra.run.smithi066.stdout: ms bind msgr2 = False 2024-03-25T18:02:56.053 INFO:teuthology.orchestra.run.smithi066.stdout: ms inject socket failures = 5000 2024-03-25T18:02:56.053 INFO:teuthology.orchestra.run.smithi066.stdout: ms type = async 2024-03-25T18:02:56.053 INFO:teuthology.orchestra.run.smithi066.stdout: osd heartbeat grace = 80 2024-03-25T18:02:56.053 INFO:teuthology.orchestra.run.smithi066.stdout: fsid = 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:02:56.053 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:56.053 INFO:teuthology.orchestra.run.smithi066.stdout:[osd] 2024-03-25T18:02:56.053 INFO:teuthology.orchestra.run.smithi066.stdout: osd journal size = 100 2024-03-25T18:02:56.053 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:56.053 INFO:teuthology.orchestra.run.smithi066.stdout: osd scrub load threshold = 5.0 2024-03-25T18:02:56.053 INFO:teuthology.orchestra.run.smithi066.stdout: osd scrub max interval = 600 2024-03-25T18:02:56.053 INFO:teuthology.orchestra.run.smithi066.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:02:56.053 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:56.053 INFO:teuthology.orchestra.run.smithi066.stdout: osd recover clone overlap = true 2024-03-25T18:02:56.053 INFO:teuthology.orchestra.run.smithi066.stdout: osd recovery max chunk = 1048576 2024-03-25T18:02:56.053 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:56.054 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug shutdown = true 2024-03-25T18:02:56.054 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug op order = true 2024-03-25T18:02:56.054 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug verify stray on activate = true 2024-03-25T18:02:56.054 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug trim objects = true 2024-03-25T18:02:56.054 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:56.054 INFO:teuthology.orchestra.run.smithi066.stdout: osd open classes on start = true 2024-03-25T18:02:56.054 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug pg log writeout = true 2024-03-25T18:02:56.054 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:56.054 INFO:teuthology.orchestra.run.smithi066.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:02:56.054 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:56.054 INFO:teuthology.orchestra.run.smithi066.stdout: osd map max advance = 10 2024-03-25T18:02:56.054 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:56.054 INFO:teuthology.orchestra.run.smithi066.stdout: journal zero on create = true 2024-03-25T18:02:56.054 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:56.055 INFO:teuthology.orchestra.run.smithi066.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:02:56.055 INFO:teuthology.orchestra.run.smithi066.stdout: filestore apply finisher threads = 3 2024-03-25T18:02:56.055 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:56.055 INFO:teuthology.orchestra.run.smithi066.stdout: bdev debug aio = true 2024-03-25T18:02:56.055 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug misdirected ops = true 2024-03-25T18:02:56.055 INFO:teuthology.orchestra.run.smithi066.stdout: bdev async discard = True 2024-03-25T18:02:56.055 INFO:teuthology.orchestra.run.smithi066.stdout: bdev enable discard = True 2024-03-25T18:02:56.055 INFO:teuthology.orchestra.run.smithi066.stdout: bluestore allocator = stupid 2024-03-25T18:02:56.055 INFO:teuthology.orchestra.run.smithi066.stdout: bluestore block size = 96636764160 2024-03-25T18:02:56.055 INFO:teuthology.orchestra.run.smithi066.stdout: bluestore fsck on mount = True 2024-03-25T18:02:56.055 INFO:teuthology.orchestra.run.smithi066.stdout: bluestore zero block detection = True 2024-03-25T18:02:56.055 INFO:teuthology.orchestra.run.smithi066.stdout: debug bluefs = 1 2024-03-25T18:02:56.055 INFO:teuthology.orchestra.run.smithi066.stdout: debug bluestore = 1 2024-03-25T18:02:56.055 INFO:teuthology.orchestra.run.smithi066.stdout: debug monc = 20 2024-03-25T18:02:56.055 INFO:teuthology.orchestra.run.smithi066.stdout: debug ms = 1 2024-03-25T18:02:56.055 INFO:teuthology.orchestra.run.smithi066.stdout: debug osd = 20 2024-03-25T18:02:56.056 INFO:teuthology.orchestra.run.smithi066.stdout: debug rocksdb = 10 2024-03-25T18:02:56.056 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:02:56.056 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd full ratio = 0.9 2024-03-25T18:02:56.056 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:02:56.056 INFO:teuthology.orchestra.run.smithi066.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:02:56.056 INFO:teuthology.orchestra.run.smithi066.stdout: osd class default list = * 2024-03-25T18:02:56.056 INFO:teuthology.orchestra.run.smithi066.stdout: osd class load list = * 2024-03-25T18:02:56.056 INFO:teuthology.orchestra.run.smithi066.stdout: osd client watch timeout = 120 2024-03-25T18:02:56.056 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug verify cached snaps = True 2024-03-25T18:02:56.056 INFO:teuthology.orchestra.run.smithi066.stdout: osd debug verify missing on start = True 2024-03-25T18:02:56.056 INFO:teuthology.orchestra.run.smithi066.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:02:56.056 INFO:teuthology.orchestra.run.smithi066.stdout: osd fast shutdown = False 2024-03-25T18:02:56.056 INFO:teuthology.orchestra.run.smithi066.stdout: osd max markdown count = 1000 2024-03-25T18:02:56.056 INFO:teuthology.orchestra.run.smithi066.stdout: osd mclock override recovery settings = True 2024-03-25T18:02:56.056 INFO:teuthology.orchestra.run.smithi066.stdout: osd objectstore = bluestore 2024-03-25T18:02:56.056 INFO:teuthology.orchestra.run.smithi066.stdout: osd op queue = debug_random 2024-03-25T18:02:56.057 INFO:teuthology.orchestra.run.smithi066.stdout: osd op queue cut off = debug_random 2024-03-25T18:02:56.057 INFO:teuthology.orchestra.run.smithi066.stdout: osd shutdown pgref assert = True 2024-03-25T18:02:56.057 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:56.057 INFO:teuthology.orchestra.run.smithi066.stdout:[mgr] 2024-03-25T18:02:56.057 INFO:teuthology.orchestra.run.smithi066.stdout: debug ms = 1 2024-03-25T18:02:56.057 INFO:teuthology.orchestra.run.smithi066.stdout: debug mgr = 20 2024-03-25T18:02:56.057 INFO:teuthology.orchestra.run.smithi066.stdout: debug mon = 20 2024-03-25T18:02:56.057 INFO:teuthology.orchestra.run.smithi066.stdout: debug auth = 20 2024-03-25T18:02:56.057 INFO:teuthology.orchestra.run.smithi066.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:02:56.057 INFO:teuthology.orchestra.run.smithi066.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:02:56.057 INFO:teuthology.orchestra.run.smithi066.stdout: mgr/telemetry/nag = false 2024-03-25T18:02:56.057 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:56.057 INFO:teuthology.orchestra.run.smithi066.stdout:[mon] 2024-03-25T18:02:56.057 INFO:teuthology.orchestra.run.smithi066.stdout: debug ms = 1 2024-03-25T18:02:56.057 INFO:teuthology.orchestra.run.smithi066.stdout: debug mon = 20 2024-03-25T18:02:56.057 INFO:teuthology.orchestra.run.smithi066.stdout: debug paxos = 20 2024-03-25T18:02:56.058 INFO:teuthology.orchestra.run.smithi066.stdout: debug auth = 20 2024-03-25T18:02:56.058 INFO:teuthology.orchestra.run.smithi066.stdout: mon data avail warn = 5 2024-03-25T18:02:56.058 INFO:teuthology.orchestra.run.smithi066.stdout: mon mgr mkfs grace = 240 2024-03-25T18:02:56.058 INFO:teuthology.orchestra.run.smithi066.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:02:56.058 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd reporter subtree level = osd 2024-03-25T18:02:56.058 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd prime pg temp = true 2024-03-25T18:02:56.058 INFO:teuthology.orchestra.run.smithi066.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:02:56.058 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:56.058 INFO:teuthology.orchestra.run.smithi066.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:02:56.058 INFO:teuthology.orchestra.run.smithi066.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:02:56.058 INFO:teuthology.orchestra.run.smithi066.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:02:56.058 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:56.059 INFO:teuthology.orchestra.run.smithi066.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:02:56.059 INFO:teuthology.orchestra.run.smithi066.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:02:56.059 INFO:teuthology.orchestra.run.smithi066.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:02:56.059 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:56.059 INFO:teuthology.orchestra.run.smithi066.stdout: # 1m isn't quite enough 2024-03-25T18:02:56.059 INFO:teuthology.orchestra.run.smithi066.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:02:56.059 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:56.059 INFO:teuthology.orchestra.run.smithi066.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:02:56.059 INFO:teuthology.orchestra.run.smithi066.stdout: mon min osdmap epochs = 50 2024-03-25T18:02:56.059 INFO:teuthology.orchestra.run.smithi066.stdout: mon osd crush smoke test = False 2024-03-25T18:02:56.059 INFO:teuthology.orchestra.run.smithi066.stdout: mon osdmap full prune interval = 2 2024-03-25T18:02:56.059 INFO:teuthology.orchestra.run.smithi066.stdout: mon osdmap full prune min = 15 2024-03-25T18:02:56.059 INFO:teuthology.orchestra.run.smithi066.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:02:56.059 INFO:teuthology.orchestra.run.smithi066.stdout: mon scrub interval = 300 2024-03-25T18:02:56.060 INFO:teuthology.orchestra.run.smithi066.stdout: mon warn on pool no app = False 2024-03-25T18:02:56.060 INFO:teuthology.orchestra.run.smithi066.stdout: paxos service trim min = 10 2024-03-25T18:02:56.060 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:02:56.060 INFO:teuthology.orchestra.run.smithi066.stdout:[client] 2024-03-25T18:02:56.060 INFO:teuthology.orchestra.run.smithi066.stdout: rgw cache enabled = true 2024-03-25T18:02:56.060 INFO:teuthology.orchestra.run.smithi066.stdout: rgw enable ops log = true 2024-03-25T18:02:56.060 INFO:teuthology.orchestra.run.smithi066.stdout: rgw enable usage log = true 2024-03-25T18:02:56.060 INFO:teuthology.orchestra.run.smithi066.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:02:56.060 INFO:teuthology.orchestra.run.smithi066.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:02:56.060 INFO:teuthology.orchestra.run.smithi066.stdout: debug monc = 20 2024-03-25T18:02:56.060 INFO:teuthology.orchestra.run.smithi066.stdout: debug ms = 1 2024-03-25T18:02:56.060 INFO:teuthology.orchestra.run.smithi066.stdout: debug objecter = 20 2024-03-25T18:02:56.060 INFO:teuthology.orchestra.run.smithi066.stdout: debug rados = 20 2024-03-25T18:02:56.060 INFO:teuthology.orchestra.run.smithi066.stdout:[mon.a] 2024-03-25T18:02:56.060 INFO:teuthology.orchestra.run.smithi066.stdout:[mon.c] 2024-03-25T18:02:56.060 INFO:teuthology.orchestra.run.smithi066.stdout:[mon.b] 2024-03-25T18:02:56.063 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T18:02:56.063 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T18:02:56.122 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T18:02:56.122 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-25T18:02:56.122 DEBUG:teuthology.orchestra.run.smithi066:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T18:02:56.137 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-25T18:02:56.137 DEBUG:teuthology.orchestra.run.smithi066:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T18:02:56.191 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi066.front.sepia.ceph.com 2024-03-25T18:02:56.192 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-25T18:02:56.192 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:02:56.192 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:02:56.278 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-25T18:02:56.278 DEBUG:teuthology.orchestra.run.smithi066:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:02:56.333 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi193.front.sepia.ceph.com 2024-03-25T18:02:56.333 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T18:02:56.334 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:02:56.334 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:02:56.374 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T18:02:56.374 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:02:56.429 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T18:02:56.429 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T18:02:56.429 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:02:56.491 INFO:teuthology.orchestra.run.smithi066.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:02:56.494 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:02:56.587 INFO:teuthology.orchestra.run.smithi193.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:02:56.589 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T18:02:56.589 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T18:02:56.590 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T18:02:56.636 INFO:teuthology.orchestra.run.smithi066.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T18:02:56.655 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T18:02:56.697 INFO:teuthology.orchestra.run.smithi193.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T18:02:56.712 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T18:02:56.712 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi066.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi193.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:02:56.712 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T18:02:56.743 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:02:56.743 INFO:tasks.ceph:role: osd.0 2024-03-25T18:02:56.743 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi066.front.sepia.ceph.com 2024-03-25T18:02:56.743 DEBUG:teuthology.orchestra.run.smithi066:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:02:56.820 INFO:teuthology.orchestra.run.smithi066.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:02:56.821 INFO:teuthology.orchestra.run.smithi066.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:02:56.821 INFO:teuthology.orchestra.run.smithi066.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:02:56.821 INFO:teuthology.orchestra.run.smithi066.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:02:56.821 INFO:teuthology.orchestra.run.smithi066.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:02:56.821 INFO:teuthology.orchestra.run.smithi066.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:02:56.821 INFO:teuthology.orchestra.run.smithi066.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:02:56.821 INFO:teuthology.orchestra.run.smithi066.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:02:56.821 INFO:teuthology.orchestra.run.smithi066.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:02:56.821 INFO:teuthology.orchestra.run.smithi066.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:02:57.327 INFO:teuthology.orchestra.run.smithi066.stdout:Discarding blocks...Done. 2024-03-25T18:02:57.329 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi066.front.sepia.ceph.com -o noatime 2024-03-25T18:02:57.329 DEBUG:teuthology.orchestra.run.smithi066:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T18:02:57.485 DEBUG:teuthology.orchestra.run.smithi066:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T18:02:57.524 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T18:02:57.597 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:02:57.597 INFO:tasks.ceph:role: osd.1 2024-03-25T18:02:57.597 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi066.front.sepia.ceph.com 2024-03-25T18:02:57.597 DEBUG:teuthology.orchestra.run.smithi066:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:02:57.675 INFO:teuthology.orchestra.run.smithi066.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:02:57.676 INFO:teuthology.orchestra.run.smithi066.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:02:57.676 INFO:teuthology.orchestra.run.smithi066.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:02:57.676 INFO:teuthology.orchestra.run.smithi066.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:02:57.676 INFO:teuthology.orchestra.run.smithi066.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:02:57.676 INFO:teuthology.orchestra.run.smithi066.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:02:57.676 INFO:teuthology.orchestra.run.smithi066.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:02:57.676 INFO:teuthology.orchestra.run.smithi066.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:02:57.676 INFO:teuthology.orchestra.run.smithi066.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:02:57.676 INFO:teuthology.orchestra.run.smithi066.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:02:58.118 INFO:teuthology.orchestra.run.smithi066.stdout:Discarding blocks...Done. 2024-03-25T18:02:58.120 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi066.front.sepia.ceph.com -o noatime 2024-03-25T18:02:58.120 DEBUG:teuthology.orchestra.run.smithi066:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T18:02:58.218 DEBUG:teuthology.orchestra.run.smithi066:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T18:02:58.256 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T18:02:58.328 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:02:58.328 INFO:tasks.ceph:role: osd.2 2024-03-25T18:02:58.328 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi066.front.sepia.ceph.com 2024-03-25T18:02:58.328 DEBUG:teuthology.orchestra.run.smithi066:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:02:58.406 INFO:teuthology.orchestra.run.smithi066.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:02:58.406 INFO:teuthology.orchestra.run.smithi066.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:02:58.406 INFO:teuthology.orchestra.run.smithi066.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:02:58.406 INFO:teuthology.orchestra.run.smithi066.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:02:58.406 INFO:teuthology.orchestra.run.smithi066.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:02:58.406 INFO:teuthology.orchestra.run.smithi066.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:02:58.406 INFO:teuthology.orchestra.run.smithi066.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:02:58.406 INFO:teuthology.orchestra.run.smithi066.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:02:58.407 INFO:teuthology.orchestra.run.smithi066.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:02:58.407 INFO:teuthology.orchestra.run.smithi066.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:02:58.891 INFO:teuthology.orchestra.run.smithi066.stdout:Discarding blocks...Done. 2024-03-25T18:02:58.893 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi066.front.sepia.ceph.com -o noatime 2024-03-25T18:02:58.893 DEBUG:teuthology.orchestra.run.smithi066:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T18:02:59.045 DEBUG:teuthology.orchestra.run.smithi066:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T18:02:59.083 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T18:02:59.154 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:02:59.155 INFO:tasks.ceph:role: osd.3 2024-03-25T18:02:59.155 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi066.front.sepia.ceph.com 2024-03-25T18:02:59.155 DEBUG:teuthology.orchestra.run.smithi066:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:02:59.233 INFO:teuthology.orchestra.run.smithi066.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:02:59.233 INFO:teuthology.orchestra.run.smithi066.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:02:59.233 INFO:teuthology.orchestra.run.smithi066.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:02:59.233 INFO:teuthology.orchestra.run.smithi066.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:02:59.233 INFO:teuthology.orchestra.run.smithi066.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:02:59.234 INFO:teuthology.orchestra.run.smithi066.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:02:59.234 INFO:teuthology.orchestra.run.smithi066.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:02:59.234 INFO:teuthology.orchestra.run.smithi066.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:02:59.234 INFO:teuthology.orchestra.run.smithi066.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:02:59.234 INFO:teuthology.orchestra.run.smithi066.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:02:59.740 INFO:teuthology.orchestra.run.smithi066.stdout:Discarding blocks...Done. 2024-03-25T18:02:59.743 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi066.front.sepia.ceph.com -o noatime 2024-03-25T18:02:59.743 DEBUG:teuthology.orchestra.run.smithi066:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T18:02:59.873 DEBUG:teuthology.orchestra.run.smithi066:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T18:02:59.911 DEBUG:teuthology.orchestra.run.smithi066:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:03:00.019 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:00.018+0000 7fb6d96a0640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T18:03:00.020 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:00.018+0000 7fb6d96a0640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T18:03:00.020 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:00.018+0000 7fb6d96a0640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:03:00.020 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:00.018+0000 7fb6d96a0640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:03:00.020 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:00.018+0000 7fb6d96a0640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:03:00.020 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:00.019+0000 7fb6d96a0640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T18:03:02.869 DEBUG:teuthology.orchestra.run.smithi066:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T18:03:02.905 DEBUG:teuthology.orchestra.run.smithi066:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:03:03.001 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:02.999+0000 7fc2d6e13640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T18:03:03.001 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:02.999+0000 7fc2d6e13640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T18:03:03.001 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:02.999+0000 7fc2d6e13640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:03:03.001 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:02.999+0000 7fc2d6e13640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:03:03.001 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:02.999+0000 7fc2d6e13640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:03:03.001 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:03.000+0000 7fc2d6e13640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T18:03:05.847 DEBUG:teuthology.orchestra.run.smithi066:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T18:03:05.882 DEBUG:teuthology.orchestra.run.smithi066:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:03:05.980 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:05.979+0000 7facbaf45640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T18:03:05.981 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:05.979+0000 7facbaf45640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T18:03:05.981 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:05.979+0000 7facbaf45640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:03:05.981 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:05.979+0000 7facbaf45640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:03:05.981 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:05.979+0000 7facbaf45640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:03:05.981 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:05.980+0000 7facbaf45640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T18:03:08.827 DEBUG:teuthology.orchestra.run.smithi066:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T18:03:08.862 DEBUG:teuthology.orchestra.run.smithi066:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:03:08.955 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:08.954+0000 7f90aaa1c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T18:03:08.956 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:08.954+0000 7f90aaa1c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T18:03:08.956 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:08.954+0000 7f90aaa1c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:03:08.956 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:08.954+0000 7f90aaa1c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:03:08.956 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:08.954+0000 7f90aaa1c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:03:08.956 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:08.955+0000 7f90aaa1c640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T18:03:11.805 DEBUG:teuthology.orchestra.run.smithi066:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T18:03:11.841 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T18:03:11.873 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:03:11.873 INFO:tasks.ceph:role: osd.4 2024-03-25T18:03:11.873 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi193.front.sepia.ceph.com 2024-03-25T18:03:11.874 DEBUG:teuthology.orchestra.run.smithi193:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:03:11.951 INFO:teuthology.orchestra.run.smithi193.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:03:11.951 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:03:11.951 INFO:teuthology.orchestra.run.smithi193.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:03:11.951 INFO:teuthology.orchestra.run.smithi193.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:03:11.951 INFO:teuthology.orchestra.run.smithi193.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:03:11.951 INFO:teuthology.orchestra.run.smithi193.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:03:11.951 INFO:teuthology.orchestra.run.smithi193.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:03:11.951 INFO:teuthology.orchestra.run.smithi193.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:03:11.951 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:03:11.952 INFO:teuthology.orchestra.run.smithi193.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:03:12.463 INFO:teuthology.orchestra.run.smithi193.stdout:Discarding blocks...Done. 2024-03-25T18:03:12.465 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi193.front.sepia.ceph.com -o noatime 2024-03-25T18:03:12.466 DEBUG:teuthology.orchestra.run.smithi193:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T18:03:12.613 DEBUG:teuthology.orchestra.run.smithi193:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T18:03:12.650 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T18:03:12.720 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:03:12.720 INFO:tasks.ceph:role: osd.5 2024-03-25T18:03:12.720 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi193.front.sepia.ceph.com 2024-03-25T18:03:12.721 DEBUG:teuthology.orchestra.run.smithi193:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:03:12.796 INFO:teuthology.orchestra.run.smithi193.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:03:12.796 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:03:12.796 INFO:teuthology.orchestra.run.smithi193.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:03:12.796 INFO:teuthology.orchestra.run.smithi193.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:03:12.796 INFO:teuthology.orchestra.run.smithi193.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:03:12.796 INFO:teuthology.orchestra.run.smithi193.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:03:12.796 INFO:teuthology.orchestra.run.smithi193.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:03:12.796 INFO:teuthology.orchestra.run.smithi193.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:03:12.796 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:03:12.796 INFO:teuthology.orchestra.run.smithi193.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:03:13.239 INFO:teuthology.orchestra.run.smithi193.stdout:Discarding blocks...Done. 2024-03-25T18:03:13.242 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi193.front.sepia.ceph.com -o noatime 2024-03-25T18:03:13.242 DEBUG:teuthology.orchestra.run.smithi193:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T18:03:13.337 DEBUG:teuthology.orchestra.run.smithi193:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T18:03:13.368 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T18:03:13.435 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:03:13.435 INFO:tasks.ceph:role: osd.6 2024-03-25T18:03:13.435 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi193.front.sepia.ceph.com 2024-03-25T18:03:13.436 DEBUG:teuthology.orchestra.run.smithi193:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:03:13.509 INFO:teuthology.orchestra.run.smithi193.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:03:13.509 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:03:13.509 INFO:teuthology.orchestra.run.smithi193.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:03:13.509 INFO:teuthology.orchestra.run.smithi193.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:03:13.509 INFO:teuthology.orchestra.run.smithi193.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:03:13.509 INFO:teuthology.orchestra.run.smithi193.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:03:13.509 INFO:teuthology.orchestra.run.smithi193.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:03:13.509 INFO:teuthology.orchestra.run.smithi193.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:03:13.510 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:03:13.510 INFO:teuthology.orchestra.run.smithi193.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:03:13.994 INFO:teuthology.orchestra.run.smithi193.stdout:Discarding blocks...Done. 2024-03-25T18:03:13.997 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi193.front.sepia.ceph.com -o noatime 2024-03-25T18:03:13.997 DEBUG:teuthology.orchestra.run.smithi193:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T18:03:14.117 DEBUG:teuthology.orchestra.run.smithi193:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T18:03:14.149 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T18:03:14.216 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:03:14.216 INFO:tasks.ceph:role: osd.7 2024-03-25T18:03:14.217 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi193.front.sepia.ceph.com 2024-03-25T18:03:14.217 DEBUG:teuthology.orchestra.run.smithi193:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:03:14.290 INFO:teuthology.orchestra.run.smithi193.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:03:14.291 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:03:14.291 INFO:teuthology.orchestra.run.smithi193.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:03:14.291 INFO:teuthology.orchestra.run.smithi193.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:03:14.291 INFO:teuthology.orchestra.run.smithi193.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:03:14.291 INFO:teuthology.orchestra.run.smithi193.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:03:14.291 INFO:teuthology.orchestra.run.smithi193.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:03:14.291 INFO:teuthology.orchestra.run.smithi193.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:03:14.291 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:03:14.291 INFO:teuthology.orchestra.run.smithi193.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:03:14.800 INFO:teuthology.orchestra.run.smithi193.stdout:Discarding blocks...Done. 2024-03-25T18:03:14.802 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi193.front.sepia.ceph.com -o noatime 2024-03-25T18:03:14.802 DEBUG:teuthology.orchestra.run.smithi193:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T18:03:14.939 DEBUG:teuthology.orchestra.run.smithi193:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T18:03:14.973 DEBUG:teuthology.orchestra.run.smithi193:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:03:15.078 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:15.077+0000 7f9bf0868640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T18:03:15.078 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:15.078+0000 7f9bf0868640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T18:03:15.078 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:15.078+0000 7f9bf0868640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:03:15.079 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:15.078+0000 7f9bf0868640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:03:15.079 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:15.078+0000 7f9bf0868640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:03:15.079 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:15.078+0000 7f9bf0868640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T18:03:17.920 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T18:03:17.952 DEBUG:teuthology.orchestra.run.smithi193:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:03:18.044 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:18.043+0000 7fdf2b015640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T18:03:18.044 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:18.043+0000 7fdf2b015640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T18:03:18.045 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:18.043+0000 7fdf2b015640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:03:18.045 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:18.043+0000 7fdf2b015640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:03:18.045 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:18.043+0000 7fdf2b015640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:03:18.045 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:18.044+0000 7fdf2b015640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T18:03:20.888 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T18:03:20.917 DEBUG:teuthology.orchestra.run.smithi193:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:03:21.007 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:21.006+0000 7f2bd5637640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T18:03:21.007 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:21.006+0000 7f2bd5637640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T18:03:21.007 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:21.006+0000 7f2bd5637640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:03:21.007 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:21.006+0000 7f2bd5637640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:03:21.007 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:21.006+0000 7f2bd5637640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:03:21.008 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:21.007+0000 7f2bd5637640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T18:03:23.848 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T18:03:23.878 DEBUG:teuthology.orchestra.run.smithi193:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:03:23.970 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:23.969+0000 7f156d697640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T18:03:23.970 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:23.969+0000 7f156d697640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T18:03:23.970 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:23.969+0000 7f156d697640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:03:23.970 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:23.969+0000 7f156d697640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:03:23.970 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:23.969+0000 7f156d697640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:03:23.971 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:23.970+0000 7f156d697640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T18:03:26.805 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T18:03:26.835 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T18:03:26.835 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-25T18:03:26.836 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T18:03:26.873 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-25T18:03:26.873 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T18:03:26.947 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-25T18:03:26.947 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T18:03:27.021 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-25T18:03:27.021 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T18:03:27.093 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-25T18:03:27.093 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T18:03:27.166 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T18:03:27.166 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T18:03:27.198 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T18:03:27.198 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T18:03:27.268 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T18:03:27.268 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T18:03:27.337 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T18:03:27.337 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T18:03:27.405 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T18:03:27.405 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T18:03:27.473 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-25T18:03:27.473 DEBUG:teuthology.orchestra.run.smithi066:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T18:03:27.490 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T18:03:27.490 DEBUG:teuthology.orchestra.run.smithi193:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T18:03:27.529 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T18:03:27.529 DEBUG:teuthology.orchestra.run.smithi066:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:03:27.532 DEBUG:teuthology.orchestra.run.smithi193:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:03:27.571 INFO:teuthology.orchestra.run.smithi066.stdout:[mgr.y] 2024-03-25T18:03:27.571 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQBQvAFm9NNaHRAAaRowbWXtAhkQJFkDDKE4Jw== 2024-03-25T18:03:27.571 INFO:teuthology.orchestra.run.smithi066.stdout:[osd.0] 2024-03-25T18:03:27.572 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQBUvAFm1cUxARAAL8MIxRecN9vxHMmoAvn/OQ== 2024-03-25T18:03:27.572 INFO:teuthology.orchestra.run.smithi066.stdout:[osd.1] 2024-03-25T18:03:27.572 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQBXvAFmRqcPABAAERzTFqxkjtATsbYq1gbKow== 2024-03-25T18:03:27.572 INFO:teuthology.orchestra.run.smithi066.stdout:[osd.2] 2024-03-25T18:03:27.572 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQBZvAFmakR5OhAAiUDQersx58KRz5BVqS0Rvg== 2024-03-25T18:03:27.572 INFO:teuthology.orchestra.run.smithi066.stdout:[osd.3] 2024-03-25T18:03:27.572 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQBcvAFmVEX3OBAAUChWnvzYaT8ZqvCYMtk24g== 2024-03-25T18:03:27.572 INFO:teuthology.orchestra.run.smithi066.stdout:[mgr.x] 2024-03-25T18:03:27.572 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQBQvAFmzXUHIxAAi5dZGUsPNZTPAZ3VV+LAEA== 2024-03-25T18:03:27.572 INFO:teuthology.orchestra.run.smithi066.stdout:[osd.4] 2024-03-25T18:03:27.572 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQBjvAFmorW1BBAAGxbsE/OgMxVOhQBnFha+2A== 2024-03-25T18:03:27.572 INFO:teuthology.orchestra.run.smithi066.stdout:[osd.5] 2024-03-25T18:03:27.572 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQBmvAFmtxirAhAAqSTxNy7qEh6LqN//enwKHQ== 2024-03-25T18:03:27.572 INFO:teuthology.orchestra.run.smithi066.stdout:[osd.6] 2024-03-25T18:03:27.572 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQBpvAFm5CdxABAAVO4aw8WPDdpPE4hrRiX/Dw== 2024-03-25T18:03:27.573 INFO:teuthology.orchestra.run.smithi066.stdout:[osd.7] 2024-03-25T18:03:27.573 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQBrvAFmOEDdORAA3cWsP6H4KDgID2p1Aojx0g== 2024-03-25T18:03:27.573 INFO:teuthology.orchestra.run.smithi066.stdout:[client.0] 2024-03-25T18:03:27.573 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQBQvAFm+Mn3JRAArgy4X7AFEIv1Pde/qwckyg== 2024-03-25T18:03:27.573 INFO:teuthology.orchestra.run.smithi066.stdout:[client.1] 2024-03-25T18:03:27.573 INFO:teuthology.orchestra.run.smithi066.stdout: key = AQBQvAFmLSuYKRAAyv9v+SM/4okjmdUl9bhOzQ== 2024-03-25T18:03:27.593 INFO:teuthology.orchestra.run.smithi193.stdout:[mgr.y] 2024-03-25T18:03:27.594 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQBQvAFm9NNaHRAAaRowbWXtAhkQJFkDDKE4Jw== 2024-03-25T18:03:27.594 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.0] 2024-03-25T18:03:27.594 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQBUvAFm1cUxARAAL8MIxRecN9vxHMmoAvn/OQ== 2024-03-25T18:03:27.594 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.1] 2024-03-25T18:03:27.594 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQBXvAFmRqcPABAAERzTFqxkjtATsbYq1gbKow== 2024-03-25T18:03:27.594 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.2] 2024-03-25T18:03:27.594 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQBZvAFmakR5OhAAiUDQersx58KRz5BVqS0Rvg== 2024-03-25T18:03:27.594 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.3] 2024-03-25T18:03:27.594 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQBcvAFmVEX3OBAAUChWnvzYaT8ZqvCYMtk24g== 2024-03-25T18:03:27.594 INFO:teuthology.orchestra.run.smithi193.stdout:[mgr.x] 2024-03-25T18:03:27.594 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQBQvAFmzXUHIxAAi5dZGUsPNZTPAZ3VV+LAEA== 2024-03-25T18:03:27.594 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.4] 2024-03-25T18:03:27.594 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQBjvAFmorW1BBAAGxbsE/OgMxVOhQBnFha+2A== 2024-03-25T18:03:27.594 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.5] 2024-03-25T18:03:27.595 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQBmvAFmtxirAhAAqSTxNy7qEh6LqN//enwKHQ== 2024-03-25T18:03:27.595 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.6] 2024-03-25T18:03:27.595 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQBpvAFm5CdxABAAVO4aw8WPDdpPE4hrRiX/Dw== 2024-03-25T18:03:27.595 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.7] 2024-03-25T18:03:27.595 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQBrvAFmOEDdORAA3cWsP6H4KDgID2p1Aojx0g== 2024-03-25T18:03:27.595 INFO:teuthology.orchestra.run.smithi193.stdout:[client.0] 2024-03-25T18:03:27.595 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQBQvAFm+Mn3JRAArgy4X7AFEIv1Pde/qwckyg== 2024-03-25T18:03:27.595 INFO:teuthology.orchestra.run.smithi193.stdout:[client.1] 2024-03-25T18:03:27.595 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQBQvAFmLSuYKRAAyv9v+SM/4okjmdUl9bhOzQ== 2024-03-25T18:03:27.596 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:03:27.616 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:03:27.689 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:03:27.691 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:03:27.746 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:03:27.748 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:03:27.803 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:03:27.805 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:03:27.859 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:03:27.861 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:03:27.914 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:03:27.916 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:03:27.970 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:03:27.972 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:03:28.025 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:03:28.027 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:03:28.082 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:03:28.084 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:03:28.138 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:03:28.140 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:03:28.194 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:03:28.196 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:03:28.252 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:03:28.255 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:03:28.311 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T18:03:28.311 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T18:03:28.341 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:03:28.442 DEBUG:teuthology.orchestra.run.smithi066:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T18:03:28.472 DEBUG:teuthology.orchestra.run.smithi066:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T18:03:28.543 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:03:28.645 DEBUG:teuthology.orchestra.run.smithi066:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T18:03:28.675 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T18:03:28.702 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:03:28.799 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T18:03:28.825 DEBUG:teuthology.orchestra.run.smithi066:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:03:28.827 DEBUG:teuthology.orchestra.run.smithi193:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:03:28.880 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T18:03:28.880 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:03:28.880 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T18:03:28.880 DEBUG:teuthology.orchestra.run.smithi066:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T18:03:28.883 INFO:tasks.ceph.mon.a:Started 2024-03-25T18:03:28.883 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:03:28.883 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T18:03:28.883 DEBUG:teuthology.orchestra.run.smithi066:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T18:03:28.887 INFO:tasks.ceph.mon.c:Started 2024-03-25T18:03:28.887 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:03:28.887 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T18:03:28.887 DEBUG:teuthology.orchestra.run.smithi193:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T18:03:28.922 INFO:tasks.ceph.mon.b:Started 2024-03-25T18:03:28.923 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T18:03:28.923 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T18:03:28.923 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T18:03:28.925 INFO:tasks.ceph.mgr.y:Started 2024-03-25T18:03:28.926 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T18:03:28.926 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T18:03:28.927 INFO:tasks.ceph.mgr.x:Started 2024-03-25T18:03:28.928 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T18:03:28.928 DEBUG:teuthology.orchestra.run.smithi066:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T18:03:29.037 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:29.035+0000 7fa21cf0e640 10 monclient: get_monmap_and_config 2024-03-25T18:03:29.037 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:29.035+0000 7fa21cf0e640 10 monclient: build_initial_monmap 2024-03-25T18:03:29.037 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:29.035+0000 7fa21cf0e640 10 monclient: monmap: 2024-03-25T18:03:29.037 INFO:teuthology.orchestra.run.smithi066.stderr:epoch 0 2024-03-25T18:03:29.037 INFO:teuthology.orchestra.run.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:29.037 INFO:teuthology.orchestra.run.smithi066.stderr:last_changed 2024-03-25T18:03:29.037391+0000 2024-03-25T18:03:29.037 INFO:teuthology.orchestra.run.smithi066.stderr:created 2024-03-25T18:03:29.037391+0000 2024-03-25T18:03:29.037 INFO:teuthology.orchestra.run.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:29.037 INFO:teuthology.orchestra.run.smithi066.stderr:election_strategy: 3 2024-03-25T18:03:29.037 INFO:teuthology.orchestra.run.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:03:29.037 INFO:teuthology.orchestra.run.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:03:29.037 INFO:teuthology.orchestra.run.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:03:29.037 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-03-25T18:03:29.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:29.036+0000 7fa21cf0e640 1 Processor -- start 2024-03-25T18:03:29.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:29.036+0000 7fa21cf0e640 1 -- start start 2024-03-25T18:03:29.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:29.036+0000 7fa21cf0e640 10 monclient: init 2024-03-25T18:03:29.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:29.037+0000 7fa21cf0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:29.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:29.037+0000 7fa21cf0e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:03:29.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:29.037+0000 7fa21cf0e640 10 monclient(hunting): picked mon.noname-a con 0x7fa2180a9920 addr v1:172.21.15.66:6789/0 2024-03-25T18:03:29.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:29.037+0000 7fa21cf0e640 10 monclient(hunting): picked mon.noname-b con 0x7fa2180b45a0 addr v1:172.21.15.66:6790/0 2024-03-25T18:03:29.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:29.037+0000 7fa21cf0e640 10 monclient(hunting): picked mon.noname-c con 0x7fa218146450 addr v1:172.21.15.193:6789/0 2024-03-25T18:03:29.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:29.037+0000 7fa21cf0e640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21806b100 con 0x7fa2180a9920 2024-03-25T18:03:29.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:29.037+0000 7fa21cf0e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2180b2940 con 0x7fa218146450 2024-03-25T18:03:29.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:29.037+0000 7fa21cf0e640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21813fec0 con 0x7fa2180b45a0 2024-03-25T18:03:29.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:29.037+0000 7fa21cf0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:29.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:29.037+0000 7fa21cf0e640 10 monclient(hunting): authenticate will time out at 6958.097458s 2024-03-25T18:03:29.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:29.037+0000 7fa21659c640 1 -- >> v1:172.21.15.66:6789/0 conn(0x7fa2180a9920 legacy=0x7fa21813f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.66:6789/0 2024-03-25T18:03:29.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:29.037+0000 7fa215d9b640 1 -- >> v1:172.21.15.66:6790/0 conn(0x7fa2180b45a0 legacy=0x7fa218145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.66:6790/0 2024-03-25T18:03:29.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:29.037+0000 7fa216d9d640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa218146450 legacy=0x7fa21814dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2024-03-25T18:03:29.239 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:29.238+0000 7fa215d9b640 1 -- >> v1:172.21.15.66:6790/0 conn(0x7fa2180b45a0 legacy=0x7fa218145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.66:6790/0 2024-03-25T18:03:29.239 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:29.238+0000 7fa21659c640 1 -- >> v1:172.21.15.66:6789/0 conn(0x7fa2180a9920 legacy=0x7fa21813f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.66:6789/0 2024-03-25T18:03:29.239 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:29.238+0000 7fa216d9d640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa218146450 legacy=0x7fa21814dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2024-03-25T18:03:29.640 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:29.639+0000 7fa215d9b640 1 -- >> v1:172.21.15.66:6790/0 conn(0x7fa2180b45a0 legacy=0x7fa218145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.66:6790/0 2024-03-25T18:03:29.640 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:29.639+0000 7fa21659c640 1 -- >> v1:172.21.15.66:6789/0 conn(0x7fa2180a9920 legacy=0x7fa21813f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.66:6789/0 2024-03-25T18:03:29.640 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:29.639+0000 7fa216d9d640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa218146450 legacy=0x7fa21814dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2024-03-25T18:03:30.441 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:30.440+0000 7fa21659c640 1 -- >> v1:172.21.15.66:6789/0 conn(0x7fa2180a9920 legacy=0x7fa21813f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.66:6789/0 2024-03-25T18:03:30.441 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:30.440+0000 7fa215d9b640 1 -- >> v1:172.21.15.66:6790/0 conn(0x7fa2180b45a0 legacy=0x7fa218145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.66:6790/0 2024-03-25T18:03:30.441 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:30.440+0000 7fa216d9d640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa218146450 legacy=0x7fa21814dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2024-03-25T18:03:32.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:32.037+0000 7fa207fff640 10 monclient(hunting): tick 2024-03-25T18:03:32.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:32.037+0000 7fa207fff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:03:32.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:32.037+0000 7fa207fff640 1 monclient(hunting): continuing hunt 2024-03-25T18:03:32.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:32.037+0000 7fa207fff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:03:32.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:32.037+0000 7fa207fff640 1 -- >> v1:172.21.15.66:6790/0 conn(0x7fa2180b45a0 legacy=0x7fa218145d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:03:32.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:32.037+0000 7fa207fff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa218146450 legacy=0x7fa21814dc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:03:32.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:32.037+0000 7fa207fff640 1 -- >> v1:172.21.15.66:6789/0 conn(0x7fa2180a9920 legacy=0x7fa21813f7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:03:32.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:32.037+0000 7fa207fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:03:32.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:32.037+0000 7fa207fff640 10 monclient(hunting): picked mon.noname-b con 0x7fa2180b45a0 addr v1:172.21.15.66:6790/0 2024-03-25T18:03:32.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:32.037+0000 7fa207fff640 10 monclient(hunting): picked mon.noname-c con 0x7fa218146450 addr v1:172.21.15.193:6789/0 2024-03-25T18:03:32.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:32.037+0000 7fa207fff640 10 monclient(hunting): picked mon.noname-a con 0x7fa2180a9920 addr v1:172.21.15.66:6789/0 2024-03-25T18:03:32.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:32.037+0000 7fa207fff640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21806b100 con 0x7fa2180a9920 2024-03-25T18:03:32.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:32.037+0000 7fa207fff640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2180b2940 con 0x7fa218146450 2024-03-25T18:03:32.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:32.037+0000 7fa207fff640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21813fec0 con 0x7fa2180b45a0 2024-03-25T18:03:32.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:32.037+0000 7fa21659c640 1 -- >> v1:172.21.15.66:6790/0 conn(0x7fa2180b45a0 legacy=0x7fa1f80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.66:6790/0 2024-03-25T18:03:32.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:32.037+0000 7fa207fff640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:32.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:32.037+0000 7fa216d9d640 1 -- >> v1:172.21.15.66:6789/0 conn(0x7fa2180a9920 legacy=0x7fa1f800c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.66:6789/0 2024-03-25T18:03:32.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:32.037+0000 7fa215d9b640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa218146450 legacy=0x7fa1f8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2024-03-25T18:03:32.239 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:32.238+0000 7fa21659c640 1 -- >> v1:172.21.15.66:6790/0 conn(0x7fa2180b45a0 legacy=0x7fa1f80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.66:6790/0 2024-03-25T18:03:32.239 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:32.238+0000 7fa216d9d640 1 -- >> v1:172.21.15.66:6789/0 conn(0x7fa2180a9920 legacy=0x7fa1f800c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.66:6789/0 2024-03-25T18:03:32.239 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:32.238+0000 7fa215d9b640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa218146450 legacy=0x7fa1f8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2024-03-25T18:03:32.640 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:32.639+0000 7fa21659c640 1 -- >> v1:172.21.15.66:6790/0 conn(0x7fa2180b45a0 legacy=0x7fa1f80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.66:6790/0 2024-03-25T18:03:32.640 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:32.639+0000 7fa216d9d640 1 -- >> v1:172.21.15.66:6789/0 conn(0x7fa2180a9920 legacy=0x7fa1f800c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.66:6789/0 2024-03-25T18:03:32.640 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:32.639+0000 7fa215d9b640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa218146450 legacy=0x7fa1f8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2024-03-25T18:03:33.441 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:33.440+0000 7fa21659c640 1 -- >> v1:172.21.15.66:6790/0 conn(0x7fa2180b45a0 legacy=0x7fa1f80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.66:6790/0 2024-03-25T18:03:33.441 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:33.440+0000 7fa216d9d640 1 -- >> v1:172.21.15.66:6789/0 conn(0x7fa2180a9920 legacy=0x7fa1f800c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.66:6789/0 2024-03-25T18:03:33.441 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:33.440+0000 7fa215d9b640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa218146450 legacy=0x7fa1f8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2024-03-25T18:03:35.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:35.037+0000 7fa207fff640 10 monclient(hunting): tick 2024-03-25T18:03:35.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:35.037+0000 7fa207fff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:03:35.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:35.037+0000 7fa207fff640 1 monclient(hunting): continuing hunt 2024-03-25T18:03:35.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:35.037+0000 7fa207fff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:03:35.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:35.037+0000 7fa207fff640 1 -- >> v1:172.21.15.66:6790/0 conn(0x7fa2180b45a0 legacy=0x7fa1f80056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:03:35.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:35.037+0000 7fa207fff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa218146450 legacy=0x7fa1f8008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:03:35.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:35.037+0000 7fa207fff640 1 -- >> v1:172.21.15.66:6789/0 conn(0x7fa2180a9920 legacy=0x7fa1f800c670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:03:35.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:35.037+0000 7fa207fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:03:35.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:35.037+0000 7fa207fff640 10 monclient(hunting): picked mon.noname-a con 0x7fa2180a9920 addr v1:172.21.15.66:6789/0 2024-03-25T18:03:35.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:35.037+0000 7fa207fff640 10 monclient(hunting): picked mon.noname-c con 0x7fa218146450 addr v1:172.21.15.193:6789/0 2024-03-25T18:03:35.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:35.037+0000 7fa207fff640 10 monclient(hunting): picked mon.noname-b con 0x7fa2180b45a0 addr v1:172.21.15.66:6790/0 2024-03-25T18:03:35.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:35.037+0000 7fa207fff640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f8010e20 con 0x7fa2180a9920 2024-03-25T18:03:35.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:35.037+0000 7fa207fff640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21806b100 con 0x7fa218146450 2024-03-25T18:03:35.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:35.037+0000 7fa207fff640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2180b2940 con 0x7fa2180b45a0 2024-03-25T18:03:35.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:35.037+0000 7fa207fff640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:35.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:35.037+0000 7fa216d9d640 1 -- >> v1:172.21.15.66:6790/0 conn(0x7fa2180b45a0 legacy=0x7fa1f80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.66:6790/0 2024-03-25T18:03:35.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:35.037+0000 7fa21659c640 1 -- >> v1:172.21.15.66:6789/0 conn(0x7fa2180a9920 legacy=0x7fa1f800c5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.66:6789/0 2024-03-25T18:03:35.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:35.037+0000 7fa215d9b640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa218146450 legacy=0x7fa1f8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2024-03-25T18:03:35.239 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:35.238+0000 7fa21659c640 1 -- >> v1:172.21.15.66:6789/0 conn(0x7fa2180a9920 legacy=0x7fa1f800c5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.66:6789/0 2024-03-25T18:03:35.239 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:35.238+0000 7fa216d9d640 1 -- >> v1:172.21.15.66:6790/0 conn(0x7fa2180b45a0 legacy=0x7fa1f80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.66:6790/0 2024-03-25T18:03:35.239 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:35.238+0000 7fa215d9b640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa218146450 legacy=0x7fa1f8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2024-03-25T18:03:35.640 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:35.639+0000 7fa21659c640 1 -- >> v1:172.21.15.66:6789/0 conn(0x7fa2180a9920 legacy=0x7fa1f800c5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.66:6789/0 2024-03-25T18:03:35.640 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:35.639+0000 7fa216d9d640 1 -- >> v1:172.21.15.66:6790/0 conn(0x7fa2180b45a0 legacy=0x7fa1f80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.66:6790/0 2024-03-25T18:03:35.640 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:35.639+0000 7fa215d9b640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa218146450 legacy=0x7fa1f8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2024-03-25T18:03:35.910 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:00:06.869 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:36.216 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:00:07.175 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:36.438 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:00:07.419 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:36.441 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:36.440+0000 7fa21659c640 1 -- >> v1:172.21.15.66:6789/0 conn(0x7fa2180a9920 legacy=0x7fa1f800c5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.66:6789/0 2024-03-25T18:03:36.441 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:36.440+0000 7fa216d9d640 1 -- >> v1:172.21.15.66:6790/0 conn(0x7fa2180b45a0 legacy=0x7fa1f80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.66:6790/0 2024-03-25T18:03:36.441 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:36.440+0000 7fa215d9b640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa218146450 legacy=0x7fa1f8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.193:6789/0 2024-03-25T18:03:36.441 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:00:07.422 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:36.767 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:00:07.747 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:36.769 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:00:07.750 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:03:38.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:38.037+0000 7fa207fff640 10 monclient(hunting): tick 2024-03-25T18:03:38.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:38.037+0000 7fa207fff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:03:38.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:38.037+0000 7fa207fff640 1 monclient(hunting): continuing hunt 2024-03-25T18:03:38.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:38.037+0000 7fa207fff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:03:38.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:38.037+0000 7fa207fff640 1 -- >> v1:172.21.15.66:6790/0 conn(0x7fa2180b45a0 legacy=0x7fa1f80056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:03:38.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:38.037+0000 7fa207fff640 1 -- >> v1:172.21.15.193:6789/0 conn(0x7fa218146450 legacy=0x7fa1f8008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:03:38.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:38.037+0000 7fa207fff640 1 -- >> v1:172.21.15.66:6789/0 conn(0x7fa2180a9920 legacy=0x7fa1f800c5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:03:38.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:38.037+0000 7fa207fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:03:38.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:38.037+0000 7fa207fff640 10 monclient(hunting): picked mon.noname-a con 0x7fa2180a9920 addr v1:172.21.15.66:6789/0 2024-03-25T18:03:38.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:38.037+0000 7fa207fff640 10 monclient(hunting): picked mon.noname-b con 0x7fa2180b45a0 addr v1:172.21.15.66:6790/0 2024-03-25T18:03:38.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:38.037+0000 7fa207fff640 10 monclient(hunting): picked mon.noname-c con 0x7fa218146450 addr v1:172.21.15.193:6789/0 2024-03-25T18:03:38.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:38.037+0000 7fa207fff640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21813fec0 con 0x7fa2180a9920 2024-03-25T18:03:38.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:38.037+0000 7fa207fff640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f8010e20 con 0x7fa218146450 2024-03-25T18:03:38.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:38.037+0000 7fa207fff640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21806b100 con 0x7fa2180b45a0 2024-03-25T18:03:38.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:38.037+0000 7fa207fff640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:38.063 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:38.062+0000 7fa216d9d640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fa218146450 0x7fa1f8008640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:47802/0 (socket says 172.21.15.66:47802) 2024-03-25T18:03:38.063 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:38.062+0000 7fa216d9d640 1 -- 172.21.15.66:0/1209292006 learned_addr learned my addr 172.21.15.66:0/1209292006 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:03:38.482 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:38.481+0000 7fa215d9b640 1 -- 172.21.15.66:0/1209292006 >> v1:172.21.15.66:6790/0 conn(0x7fa2180b45a0 legacy=0x7fa1f80056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-25T18:03:38.482 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:38.481+0000 7fa215d9b640 1 -- 172.21.15.66:0/1209292006 >> v1:172.21.15.66:6790/0 conn(0x7fa2180b45a0 legacy=0x7fa1f80056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:03:38.482 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:38.481+0000 7fa215d9b640 1 --1- 172.21.15.66:0/1209292006 >> v1:172.21.15.66:6790/0 conn(0x7fa2180b45a0 0x7fa1f80056d0 :-1 s=OPENED pgs=1 cs=1 l=1).handle_message read tag failed 2024-03-25T18:03:38.483 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:38.481+0000 7fa215d9b640 1 --1- 172.21.15.66:0/1209292006 >> v1:172.21.15.66:6790/0 conn(0x7fa2180b45a0 0x7fa1f80056d0 :-1 s=OPENED pgs=1 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:03:38.483 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:38.481+0000 7fa21559a640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.66:6790/0 2024-03-25T18:03:38.523 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:38.522+0000 7fa21659c640 1 -- 172.21.15.66:0/1209292006 >> v1:172.21.15.66:6789/0 conn(0x7fa2180a9920 legacy=0x7fa1f8005e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-25T18:03:38.523 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:38.522+0000 7fa21659c640 1 -- 172.21.15.66:0/1209292006 >> v1:172.21.15.66:6789/0 conn(0x7fa2180a9920 legacy=0x7fa1f8005e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T18:03:38.523 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:38.522+0000 7fa21659c640 1 --1- 172.21.15.66:0/1209292006 >> v1:172.21.15.66:6789/0 conn(0x7fa2180a9920 0x7fa1f8005e80 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T18:03:41.039 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:41.038+0000 7fa207fff640 10 monclient(hunting): tick 2024-03-25T18:03:41.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:41.038+0000 7fa207fff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:03:41.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:41.038+0000 7fa207fff640 1 monclient(hunting): continuing hunt 2024-03-25T18:03:41.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:41.038+0000 7fa207fff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:03:41.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:41.038+0000 7fa207fff640 1 -- 172.21.15.66:0/1209292006 >> v1:172.21.15.66:6790/0 conn(0x7fa2180b45a0 legacy=0x7fa1f80056d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:03:41.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:41.038+0000 7fa207fff640 1 -- 172.21.15.66:0/1209292006 >> v1:172.21.15.193:6789/0 conn(0x7fa218146450 legacy=0x7fa1f8008640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:41.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:41.038+0000 7fa207fff640 1 -- 172.21.15.66:0/1209292006 >> v1:172.21.15.66:6789/0 conn(0x7fa2180a9920 legacy=0x7fa1f8005e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:41.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:41.038+0000 7fa207fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:03:41.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:41.038+0000 7fa207fff640 10 monclient(hunting): picked mon.noname-b con 0x7fa2180b45a0 addr v1:172.21.15.66:6790/0 2024-03-25T18:03:41.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:41.038+0000 7fa207fff640 10 monclient(hunting): picked mon.noname-c con 0x7fa1f800c660 addr v1:172.21.15.193:6789/0 2024-03-25T18:03:41.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:41.038+0000 7fa207fff640 10 monclient(hunting): picked mon.noname-a con 0x7fa1f8012080 addr v1:172.21.15.66:6789/0 2024-03-25T18:03:41.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:41.038+0000 7fa207fff640 1 -- 172.21.15.66:0/1209292006 --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2180b2940 con 0x7fa1f8012080 2024-03-25T18:03:41.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:41.038+0000 7fa207fff640 1 -- 172.21.15.66:0/1209292006 --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f8005e80 con 0x7fa1f800c660 2024-03-25T18:03:41.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:41.038+0000 7fa207fff640 1 -- 172.21.15.66:0/1209292006 --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f80060b0 con 0x7fa2180b45a0 2024-03-25T18:03:41.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:41.038+0000 7fa207fff640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:44.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:44.039+0000 7fa207fff640 10 monclient(hunting): tick 2024-03-25T18:03:44.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:44.039+0000 7fa207fff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:03:44.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:44.039+0000 7fa207fff640 1 monclient(hunting): continuing hunt 2024-03-25T18:03:44.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:44.039+0000 7fa207fff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:03:44.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:44.039+0000 7fa207fff640 1 -- 172.21.15.66:0/1209292006 >> v1:172.21.15.66:6790/0 conn(0x7fa2180b45a0 legacy=0x7fa1f8008d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:44.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:44.039+0000 7fa207fff640 1 -- 172.21.15.66:0/1209292006 >> v1:172.21.15.193:6789/0 conn(0x7fa1f800c660 legacy=0x7fa1f80056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:44.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:44.039+0000 7fa207fff640 1 -- 172.21.15.66:0/1209292006 >> v1:172.21.15.66:6789/0 conn(0x7fa1f8012080 legacy=0x7fa1f8014470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:44.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:44.039+0000 7fa207fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:03:44.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:44.039+0000 7fa207fff640 10 monclient(hunting): picked mon.noname-b con 0x7fa1f800ce40 addr v1:172.21.15.66:6790/0 2024-03-25T18:03:44.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:44.039+0000 7fa207fff640 10 monclient(hunting): picked mon.noname-a con 0x7fa1f800d970 addr v1:172.21.15.66:6789/0 2024-03-25T18:03:44.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:44.039+0000 7fa207fff640 10 monclient(hunting): picked mon.noname-c con 0x7fa1f8016d00 addr v1:172.21.15.193:6789/0 2024-03-25T18:03:44.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:44.039+0000 7fa207fff640 1 -- 172.21.15.66:0/1209292006 --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f80062e0 con 0x7fa1f800d970 2024-03-25T18:03:44.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:44.039+0000 7fa207fff640 1 -- 172.21.15.66:0/1209292006 --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f8008dc0 con 0x7fa1f8016d00 2024-03-25T18:03:44.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:44.039+0000 7fa207fff640 1 -- 172.21.15.66:0/1209292006 --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f8008ff0 con 0x7fa1f800ce40 2024-03-25T18:03:44.042 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:44.039+0000 7fa207fff640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:47.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.039+0000 7fa207fff640 10 monclient(hunting): tick 2024-03-25T18:03:47.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.039+0000 7fa207fff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:03:47.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.039+0000 7fa207fff640 1 monclient(hunting): continuing hunt 2024-03-25T18:03:47.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.039+0000 7fa207fff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:03:47.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.039+0000 7fa207fff640 1 -- 172.21.15.66:0/1209292006 >> v1:172.21.15.66:6790/0 conn(0x7fa1f800ce40 legacy=0x7fa1f800d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:47.040 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.039+0000 7fa207fff640 1 -- 172.21.15.66:0/1209292006 >> v1:172.21.15.193:6789/0 conn(0x7fa1f8016d00 legacy=0x7fa1f8019170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:47.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.039+0000 7fa207fff640 1 -- 172.21.15.66:0/1209292006 >> v1:172.21.15.66:6789/0 conn(0x7fa1f800d970 legacy=0x7fa1f8008600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:47.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.039+0000 7fa207fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:03:47.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.039+0000 7fa207fff640 10 monclient(hunting): picked mon.noname-c con 0x7fa1f8011070 addr v1:172.21.15.193:6789/0 2024-03-25T18:03:47.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.039+0000 7fa207fff640 10 monclient(hunting): picked mon.noname-a con 0x7fa1f800d970 addr v1:172.21.15.66:6789/0 2024-03-25T18:03:47.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.039+0000 7fa207fff640 10 monclient(hunting): picked mon.noname-b con 0x7fa1f800ce40 addr v1:172.21.15.66:6790/0 2024-03-25T18:03:47.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.039+0000 7fa207fff640 1 -- 172.21.15.66:0/1209292006 --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f8009220 con 0x7fa1f800d970 2024-03-25T18:03:47.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.039+0000 7fa207fff640 1 -- 172.21.15.66:0/1209292006 --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f8011ca0 con 0x7fa1f8011070 2024-03-25T18:03:47.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.039+0000 7fa207fff640 1 -- 172.21.15.66:0/1209292006 --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f8011ed0 con 0x7fa1f800ce40 2024-03-25T18:03:47.041 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.039+0000 7fa207fff640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:47.416 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.416+0000 7fa21559a640 1 -- 172.21.15.66:0/1209292006 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3359929260 0 0) 0x7fa1f8009220 con 0x7fa1f800d970 2024-03-25T18:03:47.416 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.416+0000 7fa21559a640 10 monclient(hunting): _negotiate 2024-03-25T18:03:47.417 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.416+0000 7fa21559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:47.417 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.416+0000 7fa21559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:47.417 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.416+0000 7fa21559a640 10 monclient(hunting): my global_id is 4104 2024-03-25T18:03:47.417 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.416+0000 7fa21559a640 1 -- 172.21.15.66:0/1209292006 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa200003520 con 0x7fa1f800d970 2024-03-25T18:03:47.469 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.468+0000 7fa21559a640 1 -- 172.21.15.66:0/1209292006 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3552150618 0 0) 0x7fa200003520 con 0x7fa1f800d970 2024-03-25T18:03:47.469 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.469+0000 7fa21559a640 1 -- 172.21.15.66:0/1209292006 >> v1:172.21.15.66:6790/0 conn(0x7fa1f800ce40 legacy=0x7fa1f800d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:47.469 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.469+0000 7fa21559a640 1 -- 172.21.15.66:0/1209292006 >> v1:172.21.15.193:6789/0 conn(0x7fa1f8011070 legacy=0x7fa1f8011450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:47.469 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.469+0000 7fa21559a640 10 monclient: _finish_hunting 0 2024-03-25T18:03:47.470 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.469+0000 7fa21559a640 1 monclient: found mon.noname-a 2024-03-25T18:03:47.470 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.469+0000 7fa21559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:47.470 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.469+0000 7fa21559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.66:6789/0 2024-03-25T18:03:47.470 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.469+0000 7fa21559a640 1 -- 172.21.15.66:0/1209292006 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f8005780 con 0x7fa1f800d970 2024-03-25T18:03:47.470 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.469+0000 7fa21559a640 10 monclient: _finish_auth 0 2024-03-25T18:03:47.470 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.469+0000 7fa21559a640 10 monclient: _check_auth_tickets 2024-03-25T18:03:47.470 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.469+0000 7fa21559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:47.470 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.469+0000 7fa21cf0e640 5 monclient: authenticate success, global_id 4104 2024-03-25T18:03:47.470 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.469+0000 7fa21cf0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:47.503 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.503+0000 7fa21559a640 1 -- 172.21.15.66:0/1209292006 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fa20c003ef0 con 0x7fa1f800d970 2024-03-25T18:03:47.504 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.503+0000 7fa21559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:47.504 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.503+0000 7fa21559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:03:47.504 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.503+0000 7fa21559a640 10 monclient: dump: 2024-03-25T18:03:47.504 INFO:teuthology.orchestra.run.smithi066.stderr:epoch 1 2024-03-25T18:03:47.504 INFO:teuthology.orchestra.run.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:47.504 INFO:teuthology.orchestra.run.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:47.504 INFO:teuthology.orchestra.run.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:47.504 INFO:teuthology.orchestra.run.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:47.504 INFO:teuthology.orchestra.run.smithi066.stderr:election_strategy: 3 2024-03-25T18:03:47.504 INFO:teuthology.orchestra.run.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:47.505 INFO:teuthology.orchestra.run.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:47.505 INFO:teuthology.orchestra.run.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:47.505 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-03-25T18:03:47.505 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.503+0000 7fa21cf0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:47.512 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.512+0000 7fa21559a640 1 -- 172.21.15.66:0/1209292006 <== mon.0 v1:172.21.15.66:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa20c004200 con 0x7fa1f800d970 2024-03-25T18:03:47.512 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.512+0000 7fa21559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:47.512 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.512+0000 7fa21559a640 1 -- 172.21.15.66:0/1209292006 <== mon.0 v1:172.21.15.66:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fa20c004590 con 0x7fa1f800d970 2024-03-25T18:03:47.513 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.512+0000 7fa21559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:47.513 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.512+0000 7fa21559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:03:47.513 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.512+0000 7fa21559a640 10 monclient: dump: 2024-03-25T18:03:47.513 INFO:teuthology.orchestra.run.smithi066.stderr:epoch 1 2024-03-25T18:03:47.513 INFO:teuthology.orchestra.run.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:47.513 INFO:teuthology.orchestra.run.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:47.513 INFO:teuthology.orchestra.run.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:47.513 INFO:teuthology.orchestra.run.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:47.513 INFO:teuthology.orchestra.run.smithi066.stderr:election_strategy: 3 2024-03-25T18:03:47.513 INFO:teuthology.orchestra.run.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:47.514 INFO:teuthology.orchestra.run.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:47.514 INFO:teuthology.orchestra.run.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:47.514 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-03-25T18:03:47.514 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.512+0000 7fa21cf0e640 10 monclient: get_monmap_and_config success 2024-03-25T18:03:47.514 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.512+0000 7fa21cf0e640 4 set_mon_vals no callback set 2024-03-25T18:03:47.518 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.512+0000 7fa21cf0e640 10 monclient: shutdown 2024-03-25T18:03:47.518 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.512+0000 7fa21cf0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:47.518 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.512+0000 7fa21cf0e640 1 -- 172.21.15.66:0/1209292006 >> v1:172.21.15.66:6789/0 conn(0x7fa1f800d970 legacy=0x7fa1f8008600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:47.518 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.513+0000 7fa21cf0e640 1 -- 172.21.15.66:0/1209292006 shutdown_connections 2024-03-25T18:03:47.518 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.513+0000 7fa21cf0e640 1 -- 172.21.15.66:0/1209292006 >> 172.21.15.66:0/1209292006 conn(0x7fa2180b7f40 msgr2=0x7fa2180b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:47.518 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.513+0000 7fa21cf0e640 1 -- 172.21.15.66:0/1209292006 shutdown_connections 2024-03-25T18:03:47.519 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.513+0000 7fa21cf0e640 1 -- 172.21.15.66:0/1209292006 wait complete. 2024-03-25T18:03:47.519 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.514+0000 7fa21cf0e640 10 monclient: build_initial_monmap 2024-03-25T18:03:47.519 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.514+0000 7fa21cf0e640 10 monclient: monmap: 2024-03-25T18:03:47.519 INFO:teuthology.orchestra.run.smithi066.stderr:epoch 0 2024-03-25T18:03:47.519 INFO:teuthology.orchestra.run.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:03:47.519 INFO:teuthology.orchestra.run.smithi066.stderr:last_changed 0.000000 2024-03-25T18:03:47.519 INFO:teuthology.orchestra.run.smithi066.stderr:created 0.000000 2024-03-25T18:03:47.519 INFO:teuthology.orchestra.run.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:47.519 INFO:teuthology.orchestra.run.smithi066.stderr:election_strategy: 1 2024-03-25T18:03:47.519 INFO:teuthology.orchestra.run.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:03:47.519 INFO:teuthology.orchestra.run.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:03:47.520 INFO:teuthology.orchestra.run.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:03:47.520 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-03-25T18:03:47.520 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.514+0000 7fa21cf0e640 1 librados: starting msgr at 2024-03-25T18:03:47.520 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.514+0000 7fa21cf0e640 1 librados: starting objecter 2024-03-25T18:03:47.520 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.514+0000 7fa21cf0e640 1 Processor -- start 2024-03-25T18:03:47.520 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.514+0000 7fa21cf0e640 1 -- start start 2024-03-25T18:03:47.520 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.514+0000 7fa21cf0e640 1 librados: setting wanted keys 2024-03-25T18:03:47.520 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.514+0000 7fa21cf0e640 1 librados: calling monclient init 2024-03-25T18:03:47.520 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.514+0000 7fa21cf0e640 10 monclient: init 2024-03-25T18:03:47.520 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.515+0000 7fa21cf0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:47.521 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.515+0000 7fa21cf0e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:03:47.521 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.515+0000 7fa21cf0e640 10 monclient(hunting): picked mon.noname-a con 0x7fa1f800ce40 addr v1:172.21.15.66:6789/0 2024-03-25T18:03:47.521 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.515+0000 7fa21cf0e640 10 monclient(hunting): picked mon.noname-c con 0x7fa1f800d970 addr v1:172.21.15.66:6790/0 2024-03-25T18:03:47.521 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.515+0000 7fa21cf0e640 10 monclient(hunting): picked mon.noname-b con 0x7fa1f8011070 addr v1:172.21.15.193:6789/0 2024-03-25T18:03:47.521 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.515+0000 7fa21cf0e640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa218140f20 con 0x7fa1f800ce40 2024-03-25T18:03:47.521 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.515+0000 7fa21cf0e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa218143500 con 0x7fa1f8011070 2024-03-25T18:03:47.521 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.515+0000 7fa21cf0e640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21813ccb0 con 0x7fa1f800d970 2024-03-25T18:03:47.521 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.515+0000 7fa21cf0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:47.521 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.515+0000 7fa21cf0e640 10 monclient(hunting): authenticate will time out at 6976.574653s 2024-03-25T18:03:47.521 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.516+0000 7fa216d9d640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fa1f8011070 0x7fa21813aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:51602/0 (socket says 172.21.15.66:51602) 2024-03-25T18:03:47.521 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.516+0000 7fa216d9d640 1 -- 172.21.15.66:0/666358628 learned_addr learned my addr 172.21.15.66:0/666358628 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:03:47.522 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.516+0000 7fa216d9d640 1 -- 172.21.15.66:0/666358628 >> v1:172.21.15.193:6789/0 conn(0x7fa1f8011070 legacy=0x7fa21813aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T18:03:47.522 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.516+0000 7fa216d9d640 1 -- 172.21.15.66:0/666358628 >> v1:172.21.15.193:6789/0 conn(0x7fa1f8011070 legacy=0x7fa21813aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:03:47.522 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.516+0000 7fa216d9d640 1 --1- 172.21.15.66:0/666358628 >> v1:172.21.15.193:6789/0 conn(0x7fa1f8011070 0x7fa21813aeb0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-03-25T18:03:47.522 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.516+0000 7fa216d9d640 1 --1- 172.21.15.66:0/666358628 >> v1:172.21.15.193:6789/0 conn(0x7fa1f8011070 0x7fa21813aeb0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-25T18:03:47.522 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.516+0000 7fa206ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.193:6789/0 2024-03-25T18:03:47.522 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.517+0000 7fa206ffd640 10 client.?.objecter ms_handle_connect 0x7fa1f800d970 2024-03-25T18:03:47.522 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.517+0000 7fa206ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:47.558 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.558+0000 7fa206ffd640 10 client.?.objecter ms_handle_connect 0x7fa1f800ce40 2024-03-25T18:03:47.558 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.558+0000 7fa206ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:47.564 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.564+0000 7fa206ffd640 1 -- 172.21.15.66:0/666358628 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401850282 0 0) 0x7fa218140f20 con 0x7fa1f800ce40 2024-03-25T18:03:47.564 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.564+0000 7fa206ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:03:47.564 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.564+0000 7fa206ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:47.564 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.564+0000 7fa206ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:47.564 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.564+0000 7fa206ffd640 10 monclient(hunting): my global_id is 4110 2024-03-25T18:03:47.564 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.564+0000 7fa206ffd640 1 -- 172.21.15.66:0/666358628 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1f8009220 con 0x7fa1f800ce40 2024-03-25T18:03:47.579 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.579+0000 7fa206ffd640 1 -- 172.21.15.66:0/666358628 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1061193915 0 0) 0x7fa21813ccb0 con 0x7fa1f800d970 2024-03-25T18:03:47.580 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.579+0000 7fa206ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:03:47.580 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.579+0000 7fa206ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:47.580 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.579+0000 7fa206ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:47.580 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.579+0000 7fa206ffd640 10 monclient(hunting): my global_id is 4109 2024-03-25T18:03:47.580 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.579+0000 7fa206ffd640 1 -- 172.21.15.66:0/666358628 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa218140f20 con 0x7fa1f800d970 2024-03-25T18:03:47.596 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.596+0000 7fa206ffd640 1 -- 172.21.15.66:0/666358628 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3459084461 0 0) 0x7fa1f8009220 con 0x7fa1f800ce40 2024-03-25T18:03:47.596 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.596+0000 7fa206ffd640 1 -- 172.21.15.66:0/666358628 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa21813ccb0 con 0x7fa1f800ce40 2024-03-25T18:03:47.596 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.596+0000 7fa206ffd640 1 -- 172.21.15.66:0/666358628 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fa208004100 con 0x7fa1f800ce40 2024-03-25T18:03:47.596 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.596+0000 7fa206ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:47.596 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.596+0000 7fa206ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:03:47.596 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.596+0000 7fa206ffd640 10 monclient(hunting): dump: 2024-03-25T18:03:47.597 INFO:teuthology.orchestra.run.smithi066.stderr:epoch 1 2024-03-25T18:03:47.597 INFO:teuthology.orchestra.run.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:47.597 INFO:teuthology.orchestra.run.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:47.597 INFO:teuthology.orchestra.run.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:47.597 INFO:teuthology.orchestra.run.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:47.597 INFO:teuthology.orchestra.run.smithi066.stderr:election_strategy: 3 2024-03-25T18:03:47.597 INFO:teuthology.orchestra.run.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:47.597 INFO:teuthology.orchestra.run.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:47.597 INFO:teuthology.orchestra.run.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:47.597 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-03-25T18:03:47.597 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.596+0000 7fa206ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:03:47.653 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.652+0000 7fa206ffd640 1 -- 172.21.15.66:0/666358628 <== mon.0 v1:172.21.15.66:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1602090618 0 0) 0x7fa21813ccb0 con 0x7fa1f800ce40 2024-03-25T18:03:47.653 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.652+0000 7fa206ffd640 1 -- 172.21.15.66:0/666358628 >> v1:172.21.15.66:6790/0 conn(0x7fa1f800d970 legacy=0x7fa21813a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:47.653 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.652+0000 7fa206ffd640 1 -- 172.21.15.66:0/666358628 >> v1:172.21.15.193:6789/0 conn(0x7fa1f8011070 legacy=0x7fa21813aeb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:03:47.653 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.652+0000 7fa206ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:03:47.653 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.652+0000 7fa206ffd640 1 monclient: found mon.a 2024-03-25T18:03:47.653 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.652+0000 7fa206ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:47.654 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.652+0000 7fa206ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:03:47.654 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.652+0000 7fa206ffd640 1 -- 172.21.15.66:0/666358628 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa21813a920 con 0x7fa1f800ce40 2024-03-25T18:03:47.654 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.652+0000 7fa206ffd640 10 monclient: _finish_auth 0 2024-03-25T18:03:47.654 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.652+0000 7fa206ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:03:47.654 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.652+0000 7fa206ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:47.654 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.652+0000 7fa21cf0e640 5 monclient: authenticate success, global_id 4110 2024-03-25T18:03:47.654 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.652+0000 7fa21cf0e640 10 monclient: _renew_subs 2024-03-25T18:03:47.654 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.652+0000 7fa21cf0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:03:47.654 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.652+0000 7fa21cf0e640 1 -- 172.21.15.66:0/666358628 --> v1:172.21.15.66:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa218140890 con 0x7fa1f800ce40 2024-03-25T18:03:47.654 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.652+0000 7fa21cf0e640 10 client.4110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:03:47.654 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.652+0000 7fa21cf0e640 10 monclient: _renew_subs 2024-03-25T18:03:47.654 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.652+0000 7fa21cf0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:03:47.654 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.652+0000 7fa21cf0e640 1 -- 172.21.15.66:0/666358628 --> v1:172.21.15.66:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa218140b40 con 0x7fa1f800ce40 2024-03-25T18:03:47.655 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.652+0000 7fa21cf0e640 1 librados: init done 2024-03-25T18:03:47.655 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.653+0000 7fa21cf0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:47.655 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.653+0000 7fa21cf0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:47.655 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.653+0000 7fa21cf0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:03:47.655 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.653+0000 7fa21cf0e640 1 -- 172.21.15.66:0/666358628 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa218140cf0 con 0x7fa1f800ce40 2024-03-25T18:03:47.877 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.877+0000 7fa206ffd640 1 -- 172.21.15.66:0/666358628 <== mon.0 v1:172.21.15.66:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa208004510 con 0x7fa1f800ce40 2024-03-25T18:03:47.878 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.877+0000 7fa206ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:47.878 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.877+0000 7fa206ffd640 1 -- 172.21.15.66:0/666358628 <== mon.0 v1:172.21.15.66:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fa208004740 con 0x7fa1f800ce40 2024-03-25T18:03:47.878 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.877+0000 7fa206ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:47.878 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.877+0000 7fa206ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:03:47.878 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.877+0000 7fa206ffd640 10 monclient: dump: 2024-03-25T18:03:47.878 INFO:teuthology.orchestra.run.smithi066.stderr:epoch 1 2024-03-25T18:03:47.878 INFO:teuthology.orchestra.run.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:47.878 INFO:teuthology.orchestra.run.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:47.878 INFO:teuthology.orchestra.run.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:47.879 INFO:teuthology.orchestra.run.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:47.879 INFO:teuthology.orchestra.run.smithi066.stderr:election_strategy: 3 2024-03-25T18:03:47.879 INFO:teuthology.orchestra.run.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:47.879 INFO:teuthology.orchestra.run.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:47.879 INFO:teuthology.orchestra.run.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:47.879 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-03-25T18:03:47.879 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.877+0000 7fa206ffd640 1 -- 172.21.15.66:0/666358628 <== mon.0 v1:172.21.15.66:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa208004cf0 con 0x7fa1f800ce40 2024-03-25T18:03:47.879 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.877+0000 7fa214d99640 4 set_mon_vals no callback set 2024-03-25T18:03:47.879 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.877+0000 7fa206ffd640 1 -- 172.21.15.66:0/666358628 <== mon.0 v1:172.21.15.66:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3161724943 0 0) 0x7fa208006140 con 0x7fa1f800ce40 2024-03-25T18:03:47.879 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.877+0000 7fa206ffd640 10 client.4110.objecter ms_dispatch 0x7fa218000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T18:03:47.879 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.877+0000 7fa206ffd640 3 client.4110.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T18:03:47.880 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.877+0000 7fa206ffd640 3 client.4110.objecter handle_osd_map decoding full epoch 1 2024-03-25T18:03:47.880 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.877+0000 7fa206ffd640 20 client.4110.objecter dump_active .. 0 homeless 2024-03-25T18:03:47.880 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.877+0000 7fa206ffd640 1 -- 172.21.15.66:0/666358628 <== mon.0 v1:172.21.15.66:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa208006560 con 0x7fa1f800ce40 2024-03-25T18:03:47.880 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.877+0000 7fa206ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:47.880 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.877+0000 7fa206ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:03:47.947 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.947+0000 7fa21cf0e640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:03:47.948 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.947+0000 7fa21cf0e640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:03:47.948 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.947+0000 7fa21cf0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:03:47.948 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:47.947+0000 7fa21cf0e640 1 -- 172.21.15.66:0/666358628 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fa218141540 con 0x7fa1f800ce40 2024-03-25T18:03:48.157 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:48.157+0000 7fa206ffd640 1 -- 172.21.15.66:0/666358628 <== mon.0 v1:172.21.15.66:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fa2080052b0 con 0x7fa1f800ce40 2024-03-25T18:03:48.157 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:48.157+0000 7fa206ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:03:48.157 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:48.157+0000 7fa206ffd640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T18:03:48.157 INFO:teuthology.orchestra.run.smithi066.stderr:adjusted tunables profile to default 2024-03-25T18:03:48.159 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:48.158+0000 7fa21cf0e640 10 librados: watch_flush enter 2024-03-25T18:03:48.159 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:48.158+0000 7fa21cf0e640 10 librados: watch_flush exit 2024-03-25T18:03:48.159 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:48.158+0000 7fa21cf0e640 20 client.4110.objecter shutdown clearing up homeless session... 2024-03-25T18:03:48.159 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:48.158+0000 7fa21cf0e640 10 client.4110.objecter successfully canceled tick 2024-03-25T18:03:48.159 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:48.158+0000 7fa21cf0e640 10 monclient: shutdown 2024-03-25T18:03:48.159 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:48.158+0000 7fa21cf0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:48.159 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:48.158+0000 7fa21cf0e640 1 -- 172.21.15.66:0/666358628 >> v1:172.21.15.66:6789/0 conn(0x7fa1f800ce40 legacy=0x7fa218142df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:48.160 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:48.158+0000 7fa21cf0e640 1 -- 172.21.15.66:0/666358628 shutdown_connections 2024-03-25T18:03:48.160 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:48.158+0000 7fa21cf0e640 1 -- 172.21.15.66:0/666358628 >> 172.21.15.66:0/666358628 conn(0x7fa2180b7f40 msgr2=0x7fa2180b2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:48.160 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:48.158+0000 7fa21cf0e640 1 -- 172.21.15.66:0/666358628 shutdown_connections 2024-03-25T18:03:48.160 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:48.158+0000 7fa21cf0e640 1 -- 172.21.15.66:0/666358628 wait complete. 2024-03-25T18:03:48.160 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:48.158+0000 7fa21cf0e640 1 librados: shutdown 2024-03-25T18:03:48.170 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T18:03:48.170 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T18:03:48.170 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-25T18:03:48.170 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T18:03:48.186 INFO:tasks.ceph.mgr.y.smithi066.stderr:2024-03-25T18:03:48.186+0000 7fc1b688f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:03:48.200 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T18:03:48.200+0000 7f5e09feb040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:03:48.203 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-25T18:03:48.203 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T18:03:48.246 INFO:tasks.ceph.mgr.y.smithi066.stderr:2024-03-25T18:03:48.246+0000 7fc1b688f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:03:48.254 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T18:03:48.255+0000 7f5e09feb040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:03:48.276 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-25T18:03:48.276 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T18:03:48.348 DEBUG:teuthology.orchestra.run.smithi066:> set -ex 2024-03-25T18:03:48.348 DEBUG:teuthology.orchestra.run.smithi066:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T18:03:48.420 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T18:03:48.420 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T18:03:48.447 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T18:03:48.447 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T18:03:48.516 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T18:03:48.516 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T18:03:48.583 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-25T18:03:48.583 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T18:03:48.651 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new 508b038c-3081-4abb-aaa7-67b50c558552 0 2024-03-25T18:03:48.791 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.791+0000 7fdcead9d640 10 monclient: get_monmap_and_config 2024-03-25T18:03:48.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.791+0000 7fdcead9d640 10 monclient: build_initial_monmap 2024-03-25T18:03:48.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.792+0000 7fdcead9d640 10 monclient: monmap: 2024-03-25T18:03:48.792 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T18:03:48.792 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:48.792 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:03:48.792244+0000 2024-03-25T18:03:48.792 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:03:48.792244+0000 2024-03-25T18:03:48.792 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:48.792 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:48.793 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:03:48.793 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:03:48.793 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:03:48.793 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:48.793 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T18:03:48.792+0000 7f5e09feb040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:03:48.795 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.792+0000 7fdcead9d640 1 Processor -- start 2024-03-25T18:03:48.795 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.792+0000 7fdcead9d640 1 -- start start 2024-03-25T18:03:48.795 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.792+0000 7fdcead9d640 10 monclient: init 2024-03-25T18:03:48.795 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.793+0000 7fdcead9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:48.796 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.793+0000 7fdcead9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:03:48.796 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.793+0000 7fdcead9d640 10 monclient(hunting): picked mon.noname-a con 0x7fdce40b4700 addr v1:172.21.15.66:6789/0 2024-03-25T18:03:48.796 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.793+0000 7fdcead9d640 10 monclient(hunting): picked mon.noname-c con 0x7fdce4142a20 addr v1:172.21.15.193:6789/0 2024-03-25T18:03:48.796 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.793+0000 7fdcead9d640 10 monclient(hunting): picked mon.noname-b con 0x7fdce40a96b0 addr v1:172.21.15.66:6790/0 2024-03-25T18:03:48.796 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.793+0000 7fdcead9d640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce40a9d60 con 0x7fdce40b4700 2024-03-25T18:03:48.796 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.793+0000 7fdcead9d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce413ff10 con 0x7fdce4142a20 2024-03-25T18:03:48.796 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.793+0000 7fdcead9d640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce40aa1f0 con 0x7fdce40a96b0 2024-03-25T18:03:48.796 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.793+0000 7fdcead9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:48.796 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.793+0000 7fdcead9d640 10 monclient(hunting): authenticate will time out at 6889.126250s 2024-03-25T18:03:48.796 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.794+0000 7fdce3fff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fdce4142a20 0x7fdce413f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:47138/0 (socket says 172.21.15.193:47138) 2024-03-25T18:03:48.797 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.794+0000 7fdce3fff640 1 -- 172.21.15.193:0/3670127766 learned_addr learned my addr 172.21.15.193:0/3670127766 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T18:03:48.798 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.796+0000 7fdce37fe640 1 -- 172.21.15.193:0/3670127766 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2962282363 0 0) 0x7fdce413ff10 con 0x7fdce4142a20 2024-03-25T18:03:48.798 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.796+0000 7fdce37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:03:48.798 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.796+0000 7fdce37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:48.798 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.796+0000 7fdce37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:48.798 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.796+0000 7fdce37fe640 10 monclient(hunting): my global_id is 4111 2024-03-25T18:03:48.798 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.796+0000 7fdce37fe640 1 -- 172.21.15.193:0/3670127766 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcd4003520 con 0x7fdce4142a20 2024-03-25T18:03:48.799 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.796+0000 7fdce37fe640 1 -- 172.21.15.193:0/3670127766 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3244602312 0 0) 0x7fdce40aa1f0 con 0x7fdce40a96b0 2024-03-25T18:03:48.799 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.796+0000 7fdce37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:03:48.799 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.796+0000 7fdce37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:48.799 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.796+0000 7fdce37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:48.799 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.796+0000 7fdce37fe640 10 monclient(hunting): my global_id is 4115 2024-03-25T18:03:48.799 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.796+0000 7fdce37fe640 1 -- 172.21.15.193:0/3670127766 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdce413ff10 con 0x7fdce40a96b0 2024-03-25T18:03:48.799 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.797+0000 7fdce37fe640 1 -- 172.21.15.193:0/3670127766 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1020991763 0 0) 0x7fdce40a9d60 con 0x7fdce40b4700 2024-03-25T18:03:48.799 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.797+0000 7fdce37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:03:48.800 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.797+0000 7fdce37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:48.800 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.797+0000 7fdce37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:48.800 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.797+0000 7fdce37fe640 10 monclient(hunting): my global_id is 4116 2024-03-25T18:03:48.800 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.797+0000 7fdce37fe640 1 -- 172.21.15.193:0/3670127766 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdce40aa1f0 con 0x7fdce40b4700 2024-03-25T18:03:48.801 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.799+0000 7fdce37fe640 1 -- 172.21.15.193:0/3670127766 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2570477330 0 0) 0x7fdcd4003520 con 0x7fdce4142a20 2024-03-25T18:03:48.801 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.799+0000 7fdce37fe640 1 -- 172.21.15.193:0/3670127766 >> v1:172.21.15.66:6790/0 conn(0x7fdce40a96b0 legacy=0x7fdce413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:48.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.799+0000 7fdce37fe640 1 -- 172.21.15.193:0/3670127766 >> v1:172.21.15.66:6789/0 conn(0x7fdce40b4700 legacy=0x7fdce4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:48.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.799+0000 7fdce37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:03:48.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.799+0000 7fdce37fe640 1 monclient: found mon.noname-c 2024-03-25T18:03:48.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.799+0000 7fdce37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:48.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.799+0000 7fdce37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:03:48.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.799+0000 7fdce37fe640 1 -- 172.21.15.193:0/3670127766 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce40b4ae0 con 0x7fdce4142a20 2024-03-25T18:03:48.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.799+0000 7fdce37fe640 10 monclient: _finish_auth 0 2024-03-25T18:03:48.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.799+0000 7fdce37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:03:48.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.799+0000 7fdce37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:48.802 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.799+0000 7fdce37fe640 1 -- 172.21.15.193:0/3670127766 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fdcd0001690 con 0x7fdce4142a20 2024-03-25T18:03:48.803 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.799+0000 7fdcead9d640 5 monclient: authenticate success, global_id 4111 2024-03-25T18:03:48.803 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.799+0000 7fdcead9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:48.803 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.799+0000 7fdce37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:48.803 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.799+0000 7fdce37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:03:48.803 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.799+0000 7fdce37fe640 10 monclient: dump: 2024-03-25T18:03:48.803 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:48.803 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:48.803 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:48.803 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:48.803 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:48.803 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:48.803 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:48.804 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:48.804 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:48.804 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:48.804 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.799+0000 7fdcead9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:48.825 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.825+0000 7fdce37fe640 1 -- 172.21.15.193:0/3670127766 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcd00017f0 con 0x7fdce4142a20 2024-03-25T18:03:48.826 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.825+0000 7fdce37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:48.826 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.825+0000 7fdce37fe640 1 -- 172.21.15.193:0/3670127766 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fdcd0001c00 con 0x7fdce4142a20 2024-03-25T18:03:48.826 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.825+0000 7fdce37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:48.826 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.825+0000 7fdce37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:03:48.826 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.825+0000 7fdce37fe640 10 monclient: dump: 2024-03-25T18:03:48.826 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:48.826 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:48.826 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:48.826 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:48.826 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:48.827 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:48.827 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:48.827 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:48.827 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:48.827 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:48.827 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.825+0000 7fdcead9d640 10 monclient: get_monmap_and_config success 2024-03-25T18:03:48.828 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.825+0000 7fdcead9d640 4 set_mon_vals no callback set 2024-03-25T18:03:48.831 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.826+0000 7fdcead9d640 10 monclient: shutdown 2024-03-25T18:03:48.831 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.826+0000 7fdcead9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:48.831 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.826+0000 7fdcead9d640 1 -- 172.21.15.193:0/3670127766 >> v1:172.21.15.193:6789/0 conn(0x7fdce4142a20 legacy=0x7fdce413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:48.831 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.826+0000 7fdcead9d640 1 -- 172.21.15.193:0/3670127766 shutdown_connections 2024-03-25T18:03:48.832 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.826+0000 7fdcead9d640 1 -- 172.21.15.193:0/3670127766 >> 172.21.15.193:0/3670127766 conn(0x7fdce40668e0 msgr2=0x7fdce4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:48.832 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.826+0000 7fdcead9d640 1 -- 172.21.15.193:0/3670127766 shutdown_connections 2024-03-25T18:03:48.832 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.826+0000 7fdcead9d640 1 -- 172.21.15.193:0/3670127766 wait complete. 2024-03-25T18:03:48.832 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.827+0000 7fdcead9d640 10 monclient: build_initial_monmap 2024-03-25T18:03:48.832 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.827+0000 7fdcead9d640 10 monclient: monmap: 2024-03-25T18:03:48.832 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T18:03:48.832 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:03:48.832 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 0.000000 2024-03-25T18:03:48.832 INFO:teuthology.orchestra.run.smithi193.stderr:created 0.000000 2024-03-25T18:03:48.832 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:48.832 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 1 2024-03-25T18:03:48.833 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:03:48.833 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:03:48.833 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:03:48.833 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:48.833 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.827+0000 7fdcead9d640 1 librados: starting msgr at 2024-03-25T18:03:48.833 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.827+0000 7fdcead9d640 1 librados: starting objecter 2024-03-25T18:03:48.833 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.827+0000 7fdcead9d640 1 Processor -- start 2024-03-25T18:03:48.833 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.827+0000 7fdcead9d640 1 -- start start 2024-03-25T18:03:48.833 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.827+0000 7fdcead9d640 1 librados: setting wanted keys 2024-03-25T18:03:48.833 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.827+0000 7fdcead9d640 1 librados: calling monclient init 2024-03-25T18:03:48.833 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.827+0000 7fdcead9d640 10 monclient: init 2024-03-25T18:03:48.834 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.828+0000 7fdcead9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:48.834 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.828+0000 7fdcead9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:03:48.834 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.828+0000 7fdcead9d640 10 monclient(hunting): picked mon.noname-a con 0x7fdce40a96b0 addr v1:172.21.15.66:6789/0 2024-03-25T18:03:48.834 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.828+0000 7fdcead9d640 10 monclient(hunting): picked mon.noname-b con 0x7fdce40b4700 addr v1:172.21.15.193:6789/0 2024-03-25T18:03:48.834 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.828+0000 7fdcead9d640 10 monclient(hunting): picked mon.noname-c con 0x7fdce4142a20 addr v1:172.21.15.66:6790/0 2024-03-25T18:03:48.834 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.828+0000 7fdcead9d640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce414eca0 con 0x7fdce40a96b0 2024-03-25T18:03:48.834 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.828+0000 7fdcead9d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce41d74d0 con 0x7fdce40b4700 2024-03-25T18:03:48.834 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.828+0000 7fdcead9d640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce41d8350 con 0x7fdce4142a20 2024-03-25T18:03:48.834 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.828+0000 7fdcead9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:48.834 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.828+0000 7fdcead9d640 10 monclient(hunting): authenticate will time out at 6889.161397s 2024-03-25T18:03:48.835 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.829+0000 7fdce3fff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fdce40b4700 0x7fdce40acb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:47140/0 (socket says 172.21.15.193:47140) 2024-03-25T18:03:48.835 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.829+0000 7fdce3fff640 1 -- 172.21.15.193:0/3846761015 learned_addr learned my addr 172.21.15.193:0/3846761015 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T18:03:48.835 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.829+0000 7fdce17fa640 10 client.?.objecter ms_handle_connect 0x7fdce40b4700 2024-03-25T18:03:48.835 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.829+0000 7fdce17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:48.835 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.829+0000 7fdce17fa640 10 client.?.objecter ms_handle_connect 0x7fdce40a96b0 2024-03-25T18:03:48.835 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.829+0000 7fdce17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:48.835 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.831+0000 7fdce17fa640 1 -- 172.21.15.193:0/3846761015 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 707011278 0 0) 0x7fdce41d74d0 con 0x7fdce40b4700 2024-03-25T18:03:48.835 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.831+0000 7fdce17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:03:48.835 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.831+0000 7fdce17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:48.836 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.831+0000 7fdce17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:48.836 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.831+0000 7fdce17fa640 10 monclient(hunting): my global_id is 4114 2024-03-25T18:03:48.836 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.831+0000 7fdce17fa640 1 -- 172.21.15.193:0/3846761015 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc8004eb0 con 0x7fdce40b4700 2024-03-25T18:03:48.836 INFO:tasks.ceph.mgr.y.smithi066.stderr:2024-03-25T18:03:48.834+0000 7fc1b688f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:03:48.838 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.832+0000 7fdce17fa640 1 -- 172.21.15.193:0/3846761015 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4273452212 0 0) 0x7fdce40aa1f0 con 0x7fdce40a96b0 2024-03-25T18:03:48.838 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.832+0000 7fdce17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:03:48.838 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.832+0000 7fdce17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:48.838 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.832+0000 7fdce17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:48.839 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.832+0000 7fdce17fa640 10 monclient(hunting): my global_id is 4119 2024-03-25T18:03:48.839 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.832+0000 7fdce17fa640 1 -- 172.21.15.193:0/3846761015 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdce41d74d0 con 0x7fdce40a96b0 2024-03-25T18:03:48.839 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.836+0000 7fdce17fa640 1 -- 172.21.15.193:0/3846761015 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4077228656 0 0) 0x7fdcc8004eb0 con 0x7fdce40b4700 2024-03-25T18:03:48.839 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.836+0000 7fdce17fa640 1 -- 172.21.15.193:0/3846761015 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdce40aa1f0 con 0x7fdce40b4700 2024-03-25T18:03:48.839 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.836+0000 7fdce17fa640 1 -- 172.21.15.193:0/3846761015 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fdcd0001bc0 con 0x7fdce40b4700 2024-03-25T18:03:48.839 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.836+0000 7fdce17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:48.839 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.836+0000 7fdce17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:03:48.839 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.836+0000 7fdce17fa640 10 monclient(hunting): dump: 2024-03-25T18:03:48.839 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:48.839 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:48.840 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:48.840 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:48.840 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:48.840 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:48.840 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:48.840 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:48.840 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:48.840 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:48.840 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.836+0000 7fdce17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:03:48.840 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.838+0000 7fdce17fa640 1 -- 172.21.15.193:0/3846761015 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1387463568 0 0) 0x7fdce41d74d0 con 0x7fdce40a96b0 2024-03-25T18:03:48.840 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.838+0000 7fdce17fa640 1 -- 172.21.15.193:0/3846761015 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcc8004eb0 con 0x7fdce40a96b0 2024-03-25T18:03:48.840 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.838+0000 7fdce17fa640 1 -- 172.21.15.193:0/3846761015 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fdcd8001710 con 0x7fdce40a96b0 2024-03-25T18:03:48.841 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.838+0000 7fdce17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:48.841 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.838+0000 7fdce17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:03:48.841 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.838+0000 7fdce17fa640 10 monclient(hunting): dump: 2024-03-25T18:03:48.841 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:48.841 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:48.841 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:48.841 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:48.841 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:48.841 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:48.841 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:48.841 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:48.842 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:48.842 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:48.843 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.841+0000 7fdce17fa640 1 -- 172.21.15.193:0/3846761015 <== mon.0 v1:172.21.15.66:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3483927123 0 0) 0x7fdcc8004eb0 con 0x7fdce40a96b0 2024-03-25T18:03:48.843 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.841+0000 7fdce17fa640 1 -- 172.21.15.193:0/3846761015 >> v1:172.21.15.66:6790/0 conn(0x7fdce4142a20 legacy=0x7fdce41d4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:48.843 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.841+0000 7fdce17fa640 1 -- 172.21.15.193:0/3846761015 >> v1:172.21.15.193:6789/0 conn(0x7fdce40b4700 legacy=0x7fdce40acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:48.843 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.841+0000 7fdce17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:03:48.843 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.841+0000 7fdce17fa640 1 monclient: found mon.a 2024-03-25T18:03:48.844 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.841+0000 7fdce17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:48.844 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.841+0000 7fdce17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:03:48.844 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.841+0000 7fdce17fa640 1 -- 172.21.15.193:0/3846761015 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce40abd90 con 0x7fdce40a96b0 2024-03-25T18:03:48.844 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.841+0000 7fdce17fa640 10 monclient: _finish_auth 0 2024-03-25T18:03:48.844 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.841+0000 7fdce17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:03:48.844 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.841+0000 7fdce17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:48.844 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.841+0000 7fdcead9d640 5 monclient: authenticate success, global_id 4119 2024-03-25T18:03:48.844 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.841+0000 7fdcead9d640 10 monclient: _renew_subs 2024-03-25T18:03:48.844 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.841+0000 7fdcead9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:03:48.845 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.841+0000 7fdcead9d640 1 -- 172.21.15.193:0/3846761015 --> v1:172.21.15.66:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdce41d7720 con 0x7fdce40a96b0 2024-03-25T18:03:48.845 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.841+0000 7fdcead9d640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:03:48.845 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.841+0000 7fdcead9d640 10 monclient: _renew_subs 2024-03-25T18:03:48.845 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.841+0000 7fdcead9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:03:48.845 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.841+0000 7fdcead9d640 1 -- 172.21.15.193:0/3846761015 --> v1:172.21.15.66:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdce40ad240 con 0x7fdce40a96b0 2024-03-25T18:03:48.845 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.841+0000 7fdcead9d640 1 librados: init done 2024-03-25T18:03:48.845 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.842+0000 7fdcead9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:48.845 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.842+0000 7fdcead9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:48.845 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.842+0000 7fdcead9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:03:48.845 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.842+0000 7fdcead9d640 1 -- 172.21.15.193:0/3846761015 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdce41d78b0 con 0x7fdce40a96b0 2024-03-25T18:03:48.849 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T18:03:48.849+0000 7f5e09feb040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:03:48.893 INFO:tasks.ceph.mgr.y.smithi066.stderr:2024-03-25T18:03:48.893+0000 7fc1b688f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:03:48.894 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T18:03:48.893+0000 7f5e09feb040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:03:48.941 INFO:tasks.ceph.mgr.y.smithi066.stderr:2024-03-25T18:03:48.940+0000 7fc1b688f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:03:48.946 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T18:03:48.946+0000 7f5e09feb040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:03:48.949 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.949+0000 7fdce17fa640 1 -- 172.21.15.193:0/3846761015 <== mon.0 v1:172.21.15.66:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcd8001c20 con 0x7fdce40a96b0 2024-03-25T18:03:48.949 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.949+0000 7fdce17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:48.949 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.949+0000 7fdce17fa640 1 -- 172.21.15.193:0/3846761015 <== mon.0 v1:172.21.15.66:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fdcd80037b0 con 0x7fdce40a96b0 2024-03-25T18:03:48.949 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.949+0000 7fdce17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:48.949 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.949+0000 7fdce17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:03:48.949 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.949+0000 7fdce17fa640 10 monclient: dump: 2024-03-25T18:03:48.949 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:48.949 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:48.949 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:48.950 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:48.950 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:48.950 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:48.950 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:48.950 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:48.950 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:48.950 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:48.950 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.949+0000 7fdce17fa640 1 -- 172.21.15.193:0/3846761015 <== mon.0 v1:172.21.15.66:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fdcd8003d60 con 0x7fdce40a96b0 2024-03-25T18:03:48.950 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.949+0000 7fdce2ffd640 4 set_mon_vals no callback set 2024-03-25T18:03:48.951 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.950+0000 7fdce17fa640 1 -- 172.21.15.193:0/3846761015 <== mon.0 v1:172.21.15.66:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 3721420697 0 0) 0x7fdcd8005170 con 0x7fdce40a96b0 2024-03-25T18:03:48.951 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.950+0000 7fdce17fa640 10 client.4119.objecter ms_dispatch 0x7fdce4000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T18:03:48.951 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.950+0000 7fdce17fa640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T18:03:48.952 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.950+0000 7fdce17fa640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T18:03:48.952 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.950+0000 7fdce17fa640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T18:03:48.952 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.950+0000 7fdce17fa640 1 -- 172.21.15.193:0/3846761015 <== mon.0 v1:172.21.15.66:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fdcd8004290 con 0x7fdce40a96b0 2024-03-25T18:03:48.952 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.950+0000 7fdce17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:48.952 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:48.950+0000 7fdce17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:03:48.988 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T18:03:48.989+0000 7f5e09feb040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:03:48.998 INFO:tasks.ceph.mgr.y.smithi066.stderr:2024-03-25T18:03:48.998+0000 7fc1b688f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:03:49.020 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.020+0000 7fdcead9d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "508b038c-3081-4abb-aaa7-67b50c558552", "id": 0}] 2024-03-25T18:03:49.020 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.020+0000 7fdcead9d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "508b038c-3081-4abb-aaa7-67b50c558552", "id": 0}] 2024-03-25T18:03:49.020 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.020+0000 7fdcead9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:03:49.020 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.020+0000 7fdcead9d640 1 -- 172.21.15.193:0/3846761015 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "osd new", "uuid": "508b038c-3081-4abb-aaa7-67b50c558552", "id": 0} v 0) v1 -- 0x7fdce404f730 con 0x7fdce40a96b0 2024-03-25T18:03:49.043 INFO:tasks.ceph.mgr.y.smithi066.stderr:2024-03-25T18:03:49.043+0000 7fc1b688f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:03:49.103 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T18:03:49.104+0000 7f5e09feb040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:03:49.112 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.112+0000 7fdce17fa640 1 -- 172.21.15.193:0/3846761015 <== mon.0 v1:172.21.15.66:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "508b038c-3081-4abb-aaa7-67b50c558552", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 2131792970 0 732707515) 0x7fdcd8004640 con 0x7fdce40a96b0 2024-03-25T18:03:49.112 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.112+0000 7fdce17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "508b038c-3081-4abb-aaa7-67b50c558552", "id": 0}] 2024-03-25T18:03:49.112 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.112+0000 7fdce17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:03:49.112 INFO:teuthology.orchestra.run.smithi193.stdout:0 2024-03-25T18:03:49.113 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.113+0000 7fdcead9d640 10 librados: watch_flush enter 2024-03-25T18:03:49.113 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.113+0000 7fdcead9d640 10 librados: watch_flush exit 2024-03-25T18:03:49.113 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.113+0000 7fdcead9d640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T18:03:49.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.113+0000 7fdcead9d640 10 client.4119.objecter successfully canceled tick 2024-03-25T18:03:49.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.113+0000 7fdcead9d640 10 monclient: shutdown 2024-03-25T18:03:49.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.113+0000 7fdcead9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:49.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.113+0000 7fdcead9d640 1 -- 172.21.15.193:0/3846761015 >> v1:172.21.15.66:6789/0 conn(0x7fdce40a96b0 legacy=0x7fdce41d7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:49.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.113+0000 7fdcead9d640 1 -- 172.21.15.193:0/3846761015 shutdown_connections 2024-03-25T18:03:49.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.113+0000 7fdcead9d640 1 -- 172.21.15.193:0/3846761015 >> 172.21.15.193:0/3846761015 conn(0x7fdce40668e0 msgr2=0x7fdce40b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:49.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.113+0000 7fdcead9d640 1 -- 172.21.15.193:0/3846761015 shutdown_connections 2024-03-25T18:03:49.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.113+0000 7fdcead9d640 1 -- 172.21.15.193:0/3846761015 wait complete. 2024-03-25T18:03:49.114 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.113+0000 7fdcead9d640 1 librados: shutdown 2024-03-25T18:03:49.127 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new 6335b90f-0088-47db-932c-e693dff69998 1 2024-03-25T18:03:49.171 INFO:tasks.ceph.mgr.y.smithi066.stderr:2024-03-25T18:03:49.171+0000 7fc1b688f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:03:49.178 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T18:03:49.178+0000 7f5e09feb040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:03:49.226 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T18:03:49.227+0000 7f5e09feb040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:03:49.227 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.227+0000 7efd5a50e640 10 monclient: get_monmap_and_config 2024-03-25T18:03:49.227 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.227+0000 7efd5a50e640 10 monclient: build_initial_monmap 2024-03-25T18:03:49.227 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.227+0000 7efd5a50e640 10 monclient: monmap: 2024-03-25T18:03:49.227 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T18:03:49.227 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:49.228 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:03:49.228145+0000 2024-03-25T18:03:49.228 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:03:49.228145+0000 2024-03-25T18:03:49.228 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:49.228 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:49.228 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:03:49.228 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:03:49.228 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:03:49.228 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:49.230 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.228+0000 7efd5a50e640 1 Processor -- start 2024-03-25T18:03:49.230 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.228+0000 7efd5a50e640 1 -- start start 2024-03-25T18:03:49.230 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.228+0000 7efd5a50e640 10 monclient: init 2024-03-25T18:03:49.230 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.229+0000 7efd5a50e640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:49.230 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.229+0000 7efd5a50e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:03:49.230 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.229+0000 7efd5a50e640 10 monclient(hunting): picked mon.noname-a con 0x7efd54070210 addr v1:172.21.15.66:6789/0 2024-03-25T18:03:49.230 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.229+0000 7efd5a50e640 10 monclient(hunting): picked mon.noname-c con 0x7efd540b06d0 addr v1:172.21.15.193:6789/0 2024-03-25T18:03:49.230 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.229+0000 7efd5a50e640 10 monclient(hunting): picked mon.noname-b con 0x7efd540705f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:03:49.230 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.229+0000 7efd5a50e640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd540b0b80 con 0x7efd54070210 2024-03-25T18:03:49.230 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.229+0000 7efd5a50e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd540b0430 con 0x7efd540b06d0 2024-03-25T18:03:49.230 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.229+0000 7efd5a50e640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd54071670 con 0x7efd540705f0 2024-03-25T18:03:49.231 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.229+0000 7efd5a50e640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:49.231 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.229+0000 7efd5a50e640 10 monclient(hunting): authenticate will time out at 6889.562076s 2024-03-25T18:03:49.231 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.229+0000 7efd537fe640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7efd540b06d0 0x7efd540afd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:47150/0 (socket says 172.21.15.193:47150) 2024-03-25T18:03:49.231 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.229+0000 7efd537fe640 1 -- 172.21.15.193:0/3606410343 learned_addr learned my addr 172.21.15.193:0/3606410343 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T18:03:49.231 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.232+0000 7efd52ffd640 1 -- 172.21.15.193:0/3606410343 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4061547177 0 0) 0x7efd44001110 con 0x7efd540b06d0 2024-03-25T18:03:49.232 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.232+0000 7efd52ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:03:49.232 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.232+0000 7efd52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:49.232 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.232+0000 7efd52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:49.232 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.232+0000 7efd52ffd640 10 monclient(hunting): my global_id is 4117 2024-03-25T18:03:49.232 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.232+0000 7efd52ffd640 1 -- 172.21.15.193:0/3606410343 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd3c003520 con 0x7efd540b06d0 2024-03-25T18:03:49.233 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.232+0000 7efd52ffd640 1 -- 172.21.15.193:0/3606410343 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403862879 0 0) 0x7efd54071670 con 0x7efd540705f0 2024-03-25T18:03:49.233 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.232+0000 7efd52ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:03:49.233 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.232+0000 7efd52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:49.233 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.232+0000 7efd52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:49.233 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.232+0000 7efd52ffd640 10 monclient(hunting): my global_id is 4118 2024-03-25T18:03:49.233 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.232+0000 7efd52ffd640 1 -- 172.21.15.193:0/3606410343 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd44001110 con 0x7efd540705f0 2024-03-25T18:03:49.233 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.232+0000 7efd52ffd640 1 -- 172.21.15.193:0/3606410343 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544989665 0 0) 0x7efd540b0b80 con 0x7efd54070210 2024-03-25T18:03:49.233 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.232+0000 7efd52ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:03:49.233 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.232+0000 7efd52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:49.233 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.232+0000 7efd52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:49.234 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.232+0000 7efd52ffd640 10 monclient(hunting): my global_id is 4122 2024-03-25T18:03:49.234 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.232+0000 7efd52ffd640 1 -- 172.21.15.193:0/3606410343 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd54071670 con 0x7efd54070210 2024-03-25T18:03:49.235 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.235+0000 7efd52ffd640 1 -- 172.21.15.193:0/3606410343 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1230670573 0 0) 0x7efd3c003520 con 0x7efd540b06d0 2024-03-25T18:03:49.235 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.235+0000 7efd52ffd640 1 -- 172.21.15.193:0/3606410343 >> v1:172.21.15.66:6790/0 conn(0x7efd540705f0 legacy=0x7efd5414b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:49.235 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.235+0000 7efd52ffd640 1 -- 172.21.15.193:0/3606410343 >> v1:172.21.15.66:6789/0 conn(0x7efd54070210 legacy=0x7efd54140930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:49.235 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.235+0000 7efd52ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:03:49.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.235+0000 7efd52ffd640 1 monclient: found mon.noname-c 2024-03-25T18:03:49.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.235+0000 7efd52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:49.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.235+0000 7efd52ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:03:49.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.235+0000 7efd52ffd640 1 -- 172.21.15.193:0/3606410343 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd54070a60 con 0x7efd540b06d0 2024-03-25T18:03:49.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.235+0000 7efd52ffd640 10 monclient: _finish_auth 0 2024-03-25T18:03:49.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.235+0000 7efd52ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:03:49.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.235+0000 7efd52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:49.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.235+0000 7efd52ffd640 1 -- 172.21.15.193:0/3606410343 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7efd44001680 con 0x7efd540b06d0 2024-03-25T18:03:49.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.235+0000 7efd5a50e640 5 monclient: authenticate success, global_id 4117 2024-03-25T18:03:49.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.235+0000 7efd5a50e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:49.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.235+0000 7efd52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:49.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.235+0000 7efd52ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:03:49.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.235+0000 7efd52ffd640 10 monclient: dump: 2024-03-25T18:03:49.237 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:49.237 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:49.237 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:49.237 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:49.237 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:49.237 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:49.237 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:49.237 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:49.237 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:49.237 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:49.237 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.235+0000 7efd5a50e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:49.239 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.237+0000 7efd52ffd640 1 -- 172.21.15.193:0/3606410343 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd440017e0 con 0x7efd540b06d0 2024-03-25T18:03:49.239 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.237+0000 7efd52ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:49.239 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.237+0000 7efd52ffd640 1 -- 172.21.15.193:0/3606410343 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7efd44001b70 con 0x7efd540b06d0 2024-03-25T18:03:49.240 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.237+0000 7efd52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:49.240 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.237+0000 7efd52ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:03:49.240 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.237+0000 7efd52ffd640 10 monclient: dump: 2024-03-25T18:03:49.240 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:49.240 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:49.240 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:49.240 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:49.240 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:49.240 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:49.240 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:49.240 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:49.240 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:49.241 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:49.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.237+0000 7efd5a50e640 10 monclient: get_monmap_and_config success 2024-03-25T18:03:49.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.237+0000 7efd5a50e640 4 set_mon_vals no callback set 2024-03-25T18:03:49.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.238+0000 7efd5a50e640 10 monclient: shutdown 2024-03-25T18:03:49.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.238+0000 7efd5a50e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:49.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.238+0000 7efd5a50e640 1 -- 172.21.15.193:0/3606410343 >> v1:172.21.15.193:6789/0 conn(0x7efd540b06d0 legacy=0x7efd540afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:49.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.238+0000 7efd5a50e640 1 -- 172.21.15.193:0/3606410343 shutdown_connections 2024-03-25T18:03:49.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.238+0000 7efd5a50e640 1 -- 172.21.15.193:0/3606410343 >> 172.21.15.193:0/3606410343 conn(0x7efd540b5cd0 msgr2=0x7efd540b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:49.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.238+0000 7efd5a50e640 1 -- 172.21.15.193:0/3606410343 shutdown_connections 2024-03-25T18:03:49.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.238+0000 7efd5a50e640 1 -- 172.21.15.193:0/3606410343 wait complete. 2024-03-25T18:03:49.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.239+0000 7efd5a50e640 10 monclient: build_initial_monmap 2024-03-25T18:03:49.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.239+0000 7efd5a50e640 10 monclient: monmap: 2024-03-25T18:03:49.241 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T18:03:49.241 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:03:49.242 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 0.000000 2024-03-25T18:03:49.242 INFO:teuthology.orchestra.run.smithi193.stderr:created 0.000000 2024-03-25T18:03:49.242 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:49.242 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 1 2024-03-25T18:03:49.242 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:03:49.242 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:03:49.242 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:03:49.242 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:49.242 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.239+0000 7efd5a50e640 1 librados: starting msgr at 2024-03-25T18:03:49.242 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.239+0000 7efd5a50e640 1 librados: starting objecter 2024-03-25T18:03:49.242 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.239+0000 7efd5a50e640 1 Processor -- start 2024-03-25T18:03:49.242 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.240+0000 7efd5a50e640 1 -- start start 2024-03-25T18:03:49.242 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.240+0000 7efd5a50e640 1 librados: setting wanted keys 2024-03-25T18:03:49.242 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.240+0000 7efd5a50e640 1 librados: calling monclient init 2024-03-25T18:03:49.243 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.240+0000 7efd5a50e640 10 monclient: init 2024-03-25T18:03:49.244 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.240+0000 7efd5a50e640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:49.245 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.240+0000 7efd5a50e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:03:49.245 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.240+0000 7efd5a50e640 10 monclient(hunting): picked mon.noname-a con 0x7efd54070210 addr v1:172.21.15.66:6789/0 2024-03-25T18:03:49.245 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.240+0000 7efd5a50e640 10 monclient(hunting): picked mon.noname-b con 0x7efd540705f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:03:49.245 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.240+0000 7efd5a50e640 10 monclient(hunting): picked mon.noname-c con 0x7efd540b06d0 addr v1:172.21.15.66:6790/0 2024-03-25T18:03:49.245 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.240+0000 7efd5a50e640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd541dadb0 con 0x7efd54070210 2024-03-25T18:03:49.245 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.240+0000 7efd5a50e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd5406bcb0 con 0x7efd540705f0 2024-03-25T18:03:49.245 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.240+0000 7efd5a50e640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd5412a9f0 con 0x7efd540b06d0 2024-03-25T18:03:49.245 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.240+0000 7efd5a50e640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:49.245 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.240+0000 7efd5a50e640 10 monclient(hunting): authenticate will time out at 6889.573552s 2024-03-25T18:03:49.245 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.241+0000 7efd537fe640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7efd540705f0 0x7efd5413caa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:47162/0 (socket says 172.21.15.193:47162) 2024-03-25T18:03:49.245 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.241+0000 7efd537fe640 1 -- 172.21.15.193:0/4081889757 learned_addr learned my addr 172.21.15.193:0/4081889757 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T18:03:49.245 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.241+0000 7efd50ff9640 10 client.?.objecter ms_handle_connect 0x7efd540705f0 2024-03-25T18:03:49.246 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.241+0000 7efd50ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:49.246 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.241+0000 7efd50ff9640 10 client.?.objecter ms_handle_connect 0x7efd540b06d0 2024-03-25T18:03:49.246 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.241+0000 7efd50ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:49.246 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.241+0000 7efd50ff9640 10 client.?.objecter ms_handle_connect 0x7efd54070210 2024-03-25T18:03:49.246 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.241+0000 7efd50ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:49.246 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.243+0000 7efd50ff9640 1 -- 172.21.15.193:0/4081889757 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 304469457 0 0) 0x7efd5406bcb0 con 0x7efd540705f0 2024-03-25T18:03:49.246 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.243+0000 7efd50ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:03:49.246 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.243+0000 7efd50ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:49.246 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.243+0000 7efd50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:49.246 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.243+0000 7efd50ff9640 10 monclient(hunting): my global_id is 4120 2024-03-25T18:03:49.246 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.243+0000 7efd50ff9640 1 -- 172.21.15.193:0/4081889757 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd40004eb0 con 0x7efd540705f0 2024-03-25T18:03:49.246 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.243+0000 7efd50ff9640 1 -- 172.21.15.193:0/4081889757 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113668929 0 0) 0x7efd5412a9f0 con 0x7efd540b06d0 2024-03-25T18:03:49.246 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.243+0000 7efd50ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:03:49.246 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.243+0000 7efd50ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:49.247 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.243+0000 7efd50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:49.247 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.243+0000 7efd50ff9640 10 monclient(hunting): my global_id is 4121 2024-03-25T18:03:49.247 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.243+0000 7efd50ff9640 1 -- 172.21.15.193:0/4081889757 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd5406bcb0 con 0x7efd540b06d0 2024-03-25T18:03:49.247 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.243+0000 7efd50ff9640 1 -- 172.21.15.193:0/4081889757 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1388876354 0 0) 0x7efd541dadb0 con 0x7efd54070210 2024-03-25T18:03:49.247 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.243+0000 7efd50ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:03:49.247 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.243+0000 7efd50ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:49.247 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.243+0000 7efd50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:49.247 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.243+0000 7efd50ff9640 10 monclient(hunting): my global_id is 4125 2024-03-25T18:03:49.247 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.243+0000 7efd50ff9640 1 -- 172.21.15.193:0/4081889757 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd5412a9f0 con 0x7efd54070210 2024-03-25T18:03:49.247 INFO:tasks.ceph.mgr.y.smithi066.stderr:2024-03-25T18:03:49.248+0000 7fc1b688f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:03:49.248 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.248+0000 7efd50ff9640 1 -- 172.21.15.193:0/4081889757 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3182652844 0 0) 0x7efd40004eb0 con 0x7efd540705f0 2024-03-25T18:03:49.248 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.248+0000 7efd50ff9640 1 -- 172.21.15.193:0/4081889757 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd541dadb0 con 0x7efd540705f0 2024-03-25T18:03:49.248 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.248+0000 7efd50ff9640 1 -- 172.21.15.193:0/4081889757 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7efd440047e0 con 0x7efd540705f0 2024-03-25T18:03:49.248 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.248+0000 7efd50ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:49.248 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.248+0000 7efd50ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:03:49.249 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.248+0000 7efd50ff9640 10 monclient(hunting): dump: 2024-03-25T18:03:49.249 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:49.249 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:49.249 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:49.249 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:49.249 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:49.249 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:49.249 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:49.249 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:49.249 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:49.249 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:49.249 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.248+0000 7efd50ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:03:49.250 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.250+0000 7efd50ff9640 1 -- 172.21.15.193:0/4081889757 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3238831803 0 0) 0x7efd5412a9f0 con 0x7efd54070210 2024-03-25T18:03:49.250 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.250+0000 7efd50ff9640 1 -- 172.21.15.193:0/4081889757 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd40004eb0 con 0x7efd54070210 2024-03-25T18:03:49.250 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.250+0000 7efd50ff9640 1 -- 172.21.15.193:0/4081889757 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7efd48001830 con 0x7efd54070210 2024-03-25T18:03:49.250 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.250+0000 7efd50ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:49.250 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.250+0000 7efd50ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:03:49.250 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.250+0000 7efd50ff9640 10 monclient(hunting): dump: 2024-03-25T18:03:49.250 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:49.251 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:49.251 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:49.251 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:49.251 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:49.251 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:49.251 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:49.251 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:49.251 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:49.251 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:49.253 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.253+0000 7efd50ff9640 1 -- 172.21.15.193:0/4081889757 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2174302619 0 0) 0x7efd541dadb0 con 0x7efd540705f0 2024-03-25T18:03:49.253 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.253+0000 7efd50ff9640 1 -- 172.21.15.193:0/4081889757 >> v1:172.21.15.66:6790/0 conn(0x7efd540b06d0 legacy=0x7efd541e2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:49.253 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.253+0000 7efd50ff9640 1 -- 172.21.15.193:0/4081889757 >> v1:172.21.15.66:6789/0 conn(0x7efd54070210 legacy=0x7efd541d7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:49.253 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.253+0000 7efd50ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:03:49.253 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.253+0000 7efd50ff9640 1 monclient: found mon.b 2024-03-25T18:03:49.253 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.253+0000 7efd50ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:49.253 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.253+0000 7efd50ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:03:49.254 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.253+0000 7efd50ff9640 1 -- 172.21.15.193:0/4081889757 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd5412ab80 con 0x7efd540705f0 2024-03-25T18:03:49.254 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.253+0000 7efd50ff9640 10 monclient: _finish_auth 0 2024-03-25T18:03:49.254 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.253+0000 7efd50ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:03:49.254 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.253+0000 7efd50ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:49.254 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.253+0000 7efd5a50e640 5 monclient: authenticate success, global_id 4120 2024-03-25T18:03:49.254 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.253+0000 7efd5a50e640 10 monclient: _renew_subs 2024-03-25T18:03:49.254 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.253+0000 7efd5a50e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:03:49.254 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.253+0000 7efd5a50e640 1 -- 172.21.15.193:0/4081889757 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd541da840 con 0x7efd540705f0 2024-03-25T18:03:49.254 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.253+0000 7efd5a50e640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:03:49.254 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.253+0000 7efd5a50e640 10 monclient: _renew_subs 2024-03-25T18:03:49.254 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.253+0000 7efd5a50e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:03:49.254 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.253+0000 7efd5a50e640 1 -- 172.21.15.193:0/4081889757 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd541363d0 con 0x7efd540705f0 2024-03-25T18:03:49.254 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.253+0000 7efd5a50e640 1 librados: init done 2024-03-25T18:03:49.255 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.254+0000 7efd5a50e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:49.255 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.254+0000 7efd5a50e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:49.255 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.254+0000 7efd5a50e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:03:49.255 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.254+0000 7efd5a50e640 1 -- 172.21.15.193:0/4081889757 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd541daf40 con 0x7efd540705f0 2024-03-25T18:03:49.282 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T18:03:49.282+0000 7f5e09feb040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:03:49.299 INFO:tasks.ceph.mgr.y.smithi066.stderr:2024-03-25T18:03:49.299+0000 7fc1b688f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:03:49.330 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T18:03:49.330+0000 7f5e09feb040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:03:49.359 INFO:tasks.ceph.mgr.y.smithi066.stderr:2024-03-25T18:03:49.359+0000 7fc1b688f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:03:49.409 INFO:tasks.ceph.mgr.y.smithi066.stderr:2024-03-25T18:03:49.409+0000 7fc1b688f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:03:49.446 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.446+0000 7efd50ff9640 1 -- 172.21.15.193:0/4081889757 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd440013a0 con 0x7efd540705f0 2024-03-25T18:03:49.447 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.446+0000 7efd50ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:49.447 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.446+0000 7efd50ff9640 1 -- 172.21.15.193:0/4081889757 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7efd44004d90 con 0x7efd540705f0 2024-03-25T18:03:49.447 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.446+0000 7efd50ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:49.447 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.446+0000 7efd50ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:03:49.447 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.446+0000 7efd50ff9640 10 monclient: dump: 2024-03-25T18:03:49.447 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:49.447 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:49.447 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:49.447 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:49.447 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:49.447 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:49.447 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:49.447 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:49.447 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:49.447 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:49.448 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.446+0000 7efd50ff9640 1 -- 172.21.15.193:0/4081889757 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7efd44005280 con 0x7efd540705f0 2024-03-25T18:03:49.448 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.446+0000 7efd527fc640 4 set_mon_vals no callback set 2024-03-25T18:03:49.448 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.446+0000 7efd50ff9640 1 -- 172.21.15.193:0/4081889757 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 3555362297 0 0) 0x7efd44006750 con 0x7efd540705f0 2024-03-25T18:03:49.448 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.446+0000 7efd50ff9640 10 client.4120.objecter ms_dispatch 0x7efd54000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T18:03:49.448 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.446+0000 7efd50ff9640 3 client.4120.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T18:03:49.448 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.446+0000 7efd50ff9640 3 client.4120.objecter handle_osd_map decoding full epoch 3 2024-03-25T18:03:49.448 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.446+0000 7efd50ff9640 20 client.4120.objecter dump_active .. 0 homeless 2024-03-25T18:03:49.448 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.446+0000 7efd50ff9640 1 -- 172.21.15.193:0/4081889757 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7efd44006b90 con 0x7efd540705f0 2024-03-25T18:03:49.448 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.446+0000 7efd50ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:49.448 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.446+0000 7efd50ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:03:49.520 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.521+0000 7efd5a50e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6335b90f-0088-47db-932c-e693dff69998", "id": 1}] 2024-03-25T18:03:49.521 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.521+0000 7efd5a50e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6335b90f-0088-47db-932c-e693dff69998", "id": 1}] 2024-03-25T18:03:49.521 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.521+0000 7efd5a50e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:03:49.521 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.521+0000 7efd5a50e640 1 -- 172.21.15.193:0/4081889757 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd new", "uuid": "6335b90f-0088-47db-932c-e693dff69998", "id": 1} v 0) v1 -- 0x7efd541dabe0 con 0x7efd540705f0 2024-03-25T18:03:49.658 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.658+0000 7efd50ff9640 1 -- 172.21.15.193:0/4081889757 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6335b90f-0088-47db-932c-e693dff69998", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 3059279461 0 940483276) 0x7efd44005830 con 0x7efd540705f0 2024-03-25T18:03:49.658 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.658+0000 7efd50ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6335b90f-0088-47db-932c-e693dff69998", "id": 1}] 2024-03-25T18:03:49.658 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.658+0000 7efd50ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:03:49.658 INFO:teuthology.orchestra.run.smithi193.stdout:1 2024-03-25T18:03:49.659 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.659+0000 7efd5a50e640 10 librados: watch_flush enter 2024-03-25T18:03:49.659 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.659+0000 7efd5a50e640 10 librados: watch_flush exit 2024-03-25T18:03:49.659 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.659+0000 7efd5a50e640 20 client.4120.objecter shutdown clearing up homeless session... 2024-03-25T18:03:49.660 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.659+0000 7efd5a50e640 10 client.4120.objecter successfully canceled tick 2024-03-25T18:03:49.660 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.659+0000 7efd5a50e640 10 monclient: shutdown 2024-03-25T18:03:49.660 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.659+0000 7efd5a50e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:49.660 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.659+0000 7efd5a50e640 1 -- 172.21.15.193:0/4081889757 >> v1:172.21.15.193:6789/0 conn(0x7efd540705f0 legacy=0x7efd5413caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:49.660 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.659+0000 7efd5a50e640 1 -- 172.21.15.193:0/4081889757 shutdown_connections 2024-03-25T18:03:49.660 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.659+0000 7efd5a50e640 1 -- 172.21.15.193:0/4081889757 >> 172.21.15.193:0/4081889757 conn(0x7efd540b5cd0 msgr2=0x7efd540afc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:49.660 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.659+0000 7efd5a50e640 1 -- 172.21.15.193:0/4081889757 shutdown_connections 2024-03-25T18:03:49.660 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.659+0000 7efd5a50e640 1 -- 172.21.15.193:0/4081889757 wait complete. 2024-03-25T18:03:49.660 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.659+0000 7efd5a50e640 1 librados: shutdown 2024-03-25T18:03:49.669 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new d618ea26-b83d-46e1-bd9c-439c7d6dacc4 2 2024-03-25T18:03:49.729 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T18:03:49.729+0000 7f5e09feb040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:03:49.770 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.770+0000 7f8156b15640 10 monclient: get_monmap_and_config 2024-03-25T18:03:49.770 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.770+0000 7f8156b15640 10 monclient: build_initial_monmap 2024-03-25T18:03:49.770 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.770+0000 7f8156b15640 10 monclient: monmap: 2024-03-25T18:03:49.770 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T18:03:49.770 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:49.770 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:03:49.771010+0000 2024-03-25T18:03:49.770 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:03:49.771010+0000 2024-03-25T18:03:49.771 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:49.771 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:49.771 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:03:49.771 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:03:49.771 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:03:49.771 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:49.772 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.771+0000 7f8156b15640 1 Processor -- start 2024-03-25T18:03:49.772 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.771+0000 7f8156b15640 1 -- start start 2024-03-25T18:03:49.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.771+0000 7f8156b15640 10 monclient: init 2024-03-25T18:03:49.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.772+0000 7f8156b15640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:49.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.772+0000 7f8156b15640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:03:49.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.772+0000 7f8156b15640 10 monclient(hunting): picked mon.noname-c con 0x7f81500a9920 addr v1:172.21.15.193:6789/0 2024-03-25T18:03:49.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.772+0000 7f8156b15640 10 monclient(hunting): picked mon.noname-a con 0x7f81500b45a0 addr v1:172.21.15.66:6789/0 2024-03-25T18:03:49.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.772+0000 7f8156b15640 10 monclient(hunting): picked mon.noname-b con 0x7f8150146450 addr v1:172.21.15.66:6790/0 2024-03-25T18:03:49.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.772+0000 7f8156b15640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f815006b100 con 0x7f81500b45a0 2024-03-25T18:03:49.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.772+0000 7f8156b15640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81500b2940 con 0x7f81500a9920 2024-03-25T18:03:49.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.772+0000 7f8156b15640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f815013fec0 con 0x7f8150146450 2024-03-25T18:03:49.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.772+0000 7f8156b15640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:49.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.772+0000 7f8156b15640 10 monclient(hunting): authenticate will time out at 6890.105007s 2024-03-25T18:03:49.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.772+0000 7f81548b1640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f81500a9920 0x7f815013f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:47168/0 (socket says 172.21.15.193:47168) 2024-03-25T18:03:49.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.772+0000 7f81548b1640 1 -- 172.21.15.193:0/1122526999 learned_addr learned my addr 172.21.15.193:0/1122526999 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T18:03:49.775 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.775+0000 7f814f7fe640 1 -- 172.21.15.193:0/1122526999 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 258260754 0 0) 0x7f815013fec0 con 0x7f8150146450 2024-03-25T18:03:49.776 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.775+0000 7f814f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:03:49.776 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.775+0000 7f814f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:49.776 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.775+0000 7f814f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:49.776 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.775+0000 7f814f7fe640 10 monclient(hunting): my global_id is 4124 2024-03-25T18:03:49.776 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.775+0000 7f814f7fe640 1 -- 172.21.15.193:0/1122526999 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8140003520 con 0x7f8150146450 2024-03-25T18:03:49.776 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.775+0000 7f814f7fe640 1 -- 172.21.15.193:0/1122526999 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1339514776 0 0) 0x7f81500b2940 con 0x7f81500a9920 2024-03-25T18:03:49.776 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.775+0000 7f814f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:03:49.776 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.775+0000 7f814f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:49.776 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.775+0000 7f814f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:49.776 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.775+0000 7f814f7fe640 10 monclient(hunting): my global_id is 4123 2024-03-25T18:03:49.776 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.776+0000 7f814f7fe640 1 -- 172.21.15.193:0/1122526999 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f815013fec0 con 0x7f81500a9920 2024-03-25T18:03:49.776 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.776+0000 7f814f7fe640 1 -- 172.21.15.193:0/1122526999 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2624731045 0 0) 0x7f815006b100 con 0x7f81500b45a0 2024-03-25T18:03:49.776 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.776+0000 7f814f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:03:49.777 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.776+0000 7f814f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:49.777 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.776+0000 7f814f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:49.777 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.776+0000 7f814f7fe640 10 monclient(hunting): my global_id is 4128 2024-03-25T18:03:49.777 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.776+0000 7f814f7fe640 1 -- 172.21.15.193:0/1122526999 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81500b2940 con 0x7f81500b45a0 2024-03-25T18:03:49.779 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.779+0000 7f814f7fe640 1 -- 172.21.15.193:0/1122526999 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1982981635 0 0) 0x7f8140003520 con 0x7f8150146450 2024-03-25T18:03:49.779 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.779+0000 7f814f7fe640 1 -- 172.21.15.193:0/1122526999 >> v1:172.21.15.193:6789/0 conn(0x7f81500a9920 legacy=0x7f815013f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:49.779 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.779+0000 7f814f7fe640 1 -- 172.21.15.193:0/1122526999 >> v1:172.21.15.66:6789/0 conn(0x7f81500b45a0 legacy=0x7f8150145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:49.779 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.779+0000 7f814f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:03:49.779 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.779+0000 7f814f7fe640 1 monclient: found mon.noname-b 2024-03-25T18:03:49.779 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.779+0000 7f814f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:49.779 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.779+0000 7f814f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.66:6790/0 2024-03-25T18:03:49.779 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.779+0000 7f814f7fe640 1 -- 172.21.15.193:0/1122526999 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8150140050 con 0x7f8150146450 2024-03-25T18:03:49.779 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.779+0000 7f814f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:03:49.779 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.779+0000 7f814f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:03:49.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.779+0000 7f814f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:49.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.779+0000 7f814f7fe640 1 -- 172.21.15.193:0/1122526999 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8148001a10 con 0x7f8150146450 2024-03-25T18:03:49.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.779+0000 7f8156b15640 5 monclient: authenticate success, global_id 4124 2024-03-25T18:03:49.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.779+0000 7f8156b15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:49.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.779+0000 7f814f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:49.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.779+0000 7f814f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:03:49.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.779+0000 7f814f7fe640 10 monclient: dump: 2024-03-25T18:03:49.780 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:49.780 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:49.780 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:49.780 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:49.780 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:49.780 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:49.780 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:49.780 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:49.780 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:49.780 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:49.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.779+0000 7f8156b15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:49.805 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.806+0000 7f814f7fe640 1 -- 172.21.15.193:0/1122526999 <== mon.2 v1:172.21.15.66:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8148001b70 con 0x7f8150146450 2024-03-25T18:03:49.806 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.806+0000 7f814f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:49.806 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.806+0000 7f814f7fe640 1 -- 172.21.15.193:0/1122526999 <== mon.2 v1:172.21.15.66:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8148001e20 con 0x7f8150146450 2024-03-25T18:03:49.806 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.806+0000 7f814f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:49.806 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.806+0000 7f814f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:03:49.806 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.806+0000 7f814f7fe640 10 monclient: dump: 2024-03-25T18:03:49.806 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:49.806 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:49.806 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:49.806 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:49.806 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:49.806 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:49.806 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:49.806 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:49.807 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:49.807 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:49.807 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.806+0000 7f8156b15640 10 monclient: get_monmap_and_config success 2024-03-25T18:03:49.807 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.806+0000 7f8156b15640 4 set_mon_vals no callback set 2024-03-25T18:03:49.809 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.806+0000 7f8156b15640 10 monclient: shutdown 2024-03-25T18:03:49.810 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.806+0000 7f8156b15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:49.810 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.806+0000 7f8156b15640 1 -- 172.21.15.193:0/1122526999 >> v1:172.21.15.66:6790/0 conn(0x7f8150146450 legacy=0x7f815014dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:49.810 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.806+0000 7f8156b15640 1 -- 172.21.15.193:0/1122526999 shutdown_connections 2024-03-25T18:03:49.810 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.806+0000 7f8156b15640 1 -- 172.21.15.193:0/1122526999 >> 172.21.15.193:0/1122526999 conn(0x7f81500b7f40 msgr2=0x7f81500b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:49.810 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.806+0000 7f8156b15640 1 -- 172.21.15.193:0/1122526999 shutdown_connections 2024-03-25T18:03:49.810 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.806+0000 7f8156b15640 1 -- 172.21.15.193:0/1122526999 wait complete. 2024-03-25T18:03:49.810 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.807+0000 7f8156b15640 10 monclient: build_initial_monmap 2024-03-25T18:03:49.810 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.807+0000 7f8156b15640 10 monclient: monmap: 2024-03-25T18:03:49.810 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T18:03:49.810 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:03:49.810 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 0.000000 2024-03-25T18:03:49.810 INFO:teuthology.orchestra.run.smithi193.stderr:created 0.000000 2024-03-25T18:03:49.810 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:49.810 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 1 2024-03-25T18:03:49.811 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:03:49.811 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:03:49.811 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:03:49.811 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:49.811 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.807+0000 7f8156b15640 1 librados: starting msgr at 2024-03-25T18:03:49.811 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.807+0000 7f8156b15640 1 librados: starting objecter 2024-03-25T18:03:49.811 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.808+0000 7f8156b15640 1 Processor -- start 2024-03-25T18:03:49.811 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.808+0000 7f8156b15640 1 -- start start 2024-03-25T18:03:49.811 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.808+0000 7f8156b15640 1 librados: setting wanted keys 2024-03-25T18:03:49.811 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.808+0000 7f8156b15640 1 librados: calling monclient init 2024-03-25T18:03:49.811 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.808+0000 7f8156b15640 10 monclient: init 2024-03-25T18:03:49.811 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.808+0000 7f8156b15640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:49.811 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.808+0000 7f8156b15640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:03:49.811 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.808+0000 7f8156b15640 10 monclient(hunting): picked mon.noname-a con 0x7f81500a9920 addr v1:172.21.15.66:6789/0 2024-03-25T18:03:49.811 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.808+0000 7f8156b15640 10 monclient(hunting): picked mon.noname-b con 0x7f81500b45a0 addr v1:172.21.15.193:6789/0 2024-03-25T18:03:49.812 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.808+0000 7f8156b15640 10 monclient(hunting): picked mon.noname-c con 0x7f8150146450 addr v1:172.21.15.66:6790/0 2024-03-25T18:03:49.812 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.808+0000 7f8156b15640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8150142470 con 0x7f81500a9920 2024-03-25T18:03:49.812 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.808+0000 7f8156b15640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8150142000 con 0x7f81500b45a0 2024-03-25T18:03:49.812 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.808+0000 7f8156b15640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8150142190 con 0x7f8150146450 2024-03-25T18:03:49.812 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.808+0000 7f8156b15640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:49.812 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.808+0000 7f8156b15640 10 monclient(hunting): authenticate will time out at 6890.141629s 2024-03-25T18:03:49.812 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.809+0000 7f814ffff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f81500b45a0 0x7f81500b08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:47178/0 (socket says 172.21.15.193:47178) 2024-03-25T18:03:49.812 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.809+0000 7f814ffff640 1 -- 172.21.15.193:0/420832757 learned_addr learned my addr 172.21.15.193:0/420832757 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T18:03:49.812 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.809+0000 7f814d7fa640 10 client.?.objecter ms_handle_connect 0x7f81500b45a0 2024-03-25T18:03:49.812 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.809+0000 7f814d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:49.812 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.810+0000 7f814d7fa640 10 client.?.objecter ms_handle_connect 0x7f81500a9920 2024-03-25T18:03:49.812 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.810+0000 7f814d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:49.812 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.810+0000 7f814d7fa640 10 client.?.objecter ms_handle_connect 0x7f8150146450 2024-03-25T18:03:49.812 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.810+0000 7f814d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:49.814 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.811+0000 7f814d7fa640 1 -- 172.21.15.193:0/420832757 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177447215 0 0) 0x7f8150142000 con 0x7f81500b45a0 2024-03-25T18:03:49.814 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.811+0000 7f814d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:03:49.814 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.811+0000 7f814d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:49.814 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.811+0000 7f814d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:49.814 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.811+0000 7f814d7fa640 10 monclient(hunting): my global_id is 4126 2024-03-25T18:03:49.814 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.811+0000 7f814d7fa640 1 -- 172.21.15.193:0/420832757 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8134004eb0 con 0x7f81500b45a0 2024-03-25T18:03:49.814 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.812+0000 7f814d7fa640 1 -- 172.21.15.193:0/420832757 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863945127 0 0) 0x7f8150142470 con 0x7f81500a9920 2024-03-25T18:03:49.814 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.812+0000 7f814d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:03:49.814 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.812+0000 7f814d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:49.814 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.812+0000 7f814d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:49.814 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.812+0000 7f814d7fa640 10 monclient(hunting): my global_id is 4131 2024-03-25T18:03:49.814 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.812+0000 7f814d7fa640 1 -- 172.21.15.193:0/420832757 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8150142000 con 0x7f81500a9920 2024-03-25T18:03:49.815 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.812+0000 7f814d7fa640 1 -- 172.21.15.193:0/420832757 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4164801929 0 0) 0x7f8150142190 con 0x7f8150146450 2024-03-25T18:03:49.815 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.812+0000 7f814d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:03:49.815 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.812+0000 7f814d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:49.815 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.812+0000 7f814d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:49.815 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.812+0000 7f814d7fa640 10 monclient(hunting): my global_id is 4127 2024-03-25T18:03:49.815 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.812+0000 7f814d7fa640 1 -- 172.21.15.193:0/420832757 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8150142470 con 0x7f8150146450 2024-03-25T18:03:49.816 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.816+0000 7f814d7fa640 1 -- 172.21.15.193:0/420832757 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 722264426 0 0) 0x7f8134004eb0 con 0x7f81500b45a0 2024-03-25T18:03:49.816 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.816+0000 7f814d7fa640 1 -- 172.21.15.193:0/420832757 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8150142190 con 0x7f81500b45a0 2024-03-25T18:03:49.816 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.816+0000 7f814d7fa640 1 -- 172.21.15.193:0/420832757 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f813c001ad0 con 0x7f81500b45a0 2024-03-25T18:03:49.816 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.816+0000 7f814d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:49.816 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.816+0000 7f814d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:03:49.816 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.816+0000 7f814d7fa640 10 monclient(hunting): dump: 2024-03-25T18:03:49.816 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:49.816 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:49.816 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:49.816 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:49.816 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:49.816 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:49.817 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:49.817 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:49.817 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:49.817 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:49.817 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.816+0000 7f814d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:03:49.818 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.817+0000 7f814d7fa640 1 -- 172.21.15.193:0/420832757 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1759962923 0 0) 0x7f8150142470 con 0x7f8150146450 2024-03-25T18:03:49.818 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.817+0000 7f814d7fa640 1 -- 172.21.15.193:0/420832757 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8134004eb0 con 0x7f8150146450 2024-03-25T18:03:49.818 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.817+0000 7f814d7fa640 1 -- 172.21.15.193:0/420832757 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8148001e80 con 0x7f8150146450 2024-03-25T18:03:49.818 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.817+0000 7f814d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:49.818 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.817+0000 7f814d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:03:49.818 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.817+0000 7f814d7fa640 10 monclient(hunting): dump: 2024-03-25T18:03:49.818 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:49.819 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:49.819 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:49.819 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:49.819 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:49.819 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:49.819 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:49.819 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:49.819 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:49.819 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:49.819 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.818+0000 7f814d7fa640 1 -- 172.21.15.193:0/420832757 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3516252821 0 0) 0x7f8150142000 con 0x7f81500a9920 2024-03-25T18:03:49.819 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.818+0000 7f814d7fa640 1 -- 172.21.15.193:0/420832757 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8150142470 con 0x7f81500a9920 2024-03-25T18:03:49.819 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.818+0000 7f814d7fa640 1 -- 172.21.15.193:0/420832757 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f81440018f0 con 0x7f81500a9920 2024-03-25T18:03:49.819 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.818+0000 7f814d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:49.820 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.818+0000 7f814d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:03:49.820 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.818+0000 7f814d7fa640 10 monclient(hunting): dump: 2024-03-25T18:03:49.820 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:49.820 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:49.820 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:49.820 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:49.820 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:49.820 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:49.820 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:49.820 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:49.820 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:49.820 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:49.822 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.820+0000 7f814d7fa640 1 -- 172.21.15.193:0/420832757 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 832858106 0 0) 0x7f8150142190 con 0x7f81500b45a0 2024-03-25T18:03:49.822 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.820+0000 7f814d7fa640 1 -- 172.21.15.193:0/420832757 >> v1:172.21.15.66:6790/0 conn(0x7f8150146450 legacy=0x7f81500b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:49.822 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.820+0000 7f814d7fa640 1 -- 172.21.15.193:0/420832757 >> v1:172.21.15.66:6789/0 conn(0x7f81500a9920 legacy=0x7f81501d7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:49.822 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.820+0000 7f814d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:03:49.822 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.820+0000 7f814d7fa640 1 monclient: found mon.b 2024-03-25T18:03:49.822 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.820+0000 7f814d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:49.822 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.820+0000 7f814d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:03:49.822 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.820+0000 7f814d7fa640 1 -- 172.21.15.193:0/420832757 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81500b16d0 con 0x7f81500b45a0 2024-03-25T18:03:49.822 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.820+0000 7f814d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:03:49.823 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.820+0000 7f814d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:03:49.823 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.820+0000 7f814d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:49.823 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.820+0000 7f8156b15640 5 monclient: authenticate success, global_id 4126 2024-03-25T18:03:49.823 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.820+0000 7f8156b15640 10 monclient: _renew_subs 2024-03-25T18:03:49.823 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.820+0000 7f8156b15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:03:49.823 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.820+0000 7f8156b15640 1 -- 172.21.15.193:0/420832757 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f815013bf40 con 0x7f81500b45a0 2024-03-25T18:03:49.823 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.820+0000 7f8156b15640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:03:49.823 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.820+0000 7f8156b15640 10 monclient: _renew_subs 2024-03-25T18:03:49.823 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.820+0000 7f8156b15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:03:49.823 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.820+0000 7f8156b15640 1 -- 172.21.15.193:0/420832757 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8150141bf0 con 0x7f81500b45a0 2024-03-25T18:03:49.823 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.820+0000 7f8156b15640 1 librados: init done 2024-03-25T18:03:49.823 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.821+0000 7f8156b15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:49.823 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.821+0000 7f8156b15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:49.823 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.821+0000 7f8156b15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:03:49.824 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.821+0000 7f8156b15640 1 -- 172.21.15.193:0/420832757 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f815004f730 con 0x7f81500b45a0 2024-03-25T18:03:49.828 INFO:tasks.ceph.mgr.y.smithi066.stderr:2024-03-25T18:03:49.828+0000 7fc1b688f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:03:49.881 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T18:03:49.881+0000 7f5e09feb040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:03:49.929 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.929+0000 7f814d7fa640 1 -- 172.21.15.193:0/420832757 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f813c001c30 con 0x7f81500b45a0 2024-03-25T18:03:49.929 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.929+0000 7f814d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:49.929 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.929+0000 7f814d7fa640 1 -- 172.21.15.193:0/420832757 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f813c003a40 con 0x7f81500b45a0 2024-03-25T18:03:49.929 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.929+0000 7f814d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:49.929 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.929+0000 7f814d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:03:49.929 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.929+0000 7f814d7fa640 10 monclient: dump: 2024-03-25T18:03:49.929 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:49.929 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:49.929 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:49.930 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:49.930 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:49.930 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:49.930 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:49.930 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:49.930 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:49.930 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:49.930 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.929+0000 7f814d7fa640 1 -- 172.21.15.193:0/420832757 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f813c003fb0 con 0x7f81500b45a0 2024-03-25T18:03:49.930 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.929+0000 7f814effd640 4 set_mon_vals no callback set 2024-03-25T18:03:49.930 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.929+0000 7f814d7fa640 1 -- 172.21.15.193:0/420832757 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 1860739522 0 0) 0x7f813c006650 con 0x7f81500b45a0 2024-03-25T18:03:49.930 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.929+0000 7f814d7fa640 10 client.4126.objecter ms_dispatch 0x7f8150000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T18:03:49.930 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.929+0000 7f814d7fa640 3 client.4126.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T18:03:49.930 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.929+0000 7f814d7fa640 3 client.4126.objecter handle_osd_map decoding full epoch 4 2024-03-25T18:03:49.930 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.929+0000 7f814d7fa640 20 client.4126.objecter dump_active .. 0 homeless 2024-03-25T18:03:49.931 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.929+0000 7f814d7fa640 1 -- 172.21.15.193:0/420832757 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f813c006a70 con 0x7f81500b45a0 2024-03-25T18:03:49.931 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.929+0000 7f814d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:49.931 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:49.929+0000 7f814d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:03:49.970 INFO:tasks.ceph.mgr.y.smithi066.stderr:2024-03-25T18:03:49.970+0000 7fc1b688f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:03:49.999 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T18:03:49.999+0000 7f5e09feb040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:03:49.999 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.000+0000 7f8156b15640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d618ea26-b83d-46e1-bd9c-439c7d6dacc4", "id": 2}] 2024-03-25T18:03:50.000 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.000+0000 7f8156b15640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d618ea26-b83d-46e1-bd9c-439c7d6dacc4", "id": 2}] 2024-03-25T18:03:50.000 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.000+0000 7f8156b15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:03:50.000 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.000+0000 7f8156b15640 1 -- 172.21.15.193:0/420832757 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d618ea26-b83d-46e1-bd9c-439c7d6dacc4", "id": 2} v 0) v1 -- 0x7f8150144e00 con 0x7f81500b45a0 2024-03-25T18:03:50.046 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T18:03:50.046+0000 7f5e09feb040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:03:50.102 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.103+0000 7f814d7fa640 1 -- 172.21.15.193:0/420832757 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d618ea26-b83d-46e1-bd9c-439c7d6dacc4", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 774363964 0 216599125) 0x7f813c006230 con 0x7f81500b45a0 2024-03-25T18:03:50.102 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.103+0000 7f814d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d618ea26-b83d-46e1-bd9c-439c7d6dacc4", "id": 2}] 2024-03-25T18:03:50.102 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.103+0000 7f814d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:03:50.103 INFO:teuthology.orchestra.run.smithi193.stdout:2 2024-03-25T18:03:50.104 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.104+0000 7f8156b15640 10 librados: watch_flush enter 2024-03-25T18:03:50.104 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.104+0000 7f8156b15640 10 librados: watch_flush exit 2024-03-25T18:03:50.104 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.104+0000 7f8156b15640 20 client.4126.objecter shutdown clearing up homeless session... 2024-03-25T18:03:50.104 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.104+0000 7f8156b15640 10 client.4126.objecter successfully canceled tick 2024-03-25T18:03:50.104 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.104+0000 7f8156b15640 10 monclient: shutdown 2024-03-25T18:03:50.105 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.104+0000 7f8156b15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:50.105 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.104+0000 7f8156b15640 1 -- 172.21.15.193:0/420832757 >> v1:172.21.15.193:6789/0 conn(0x7f81500b45a0 legacy=0x7f81500b08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:50.105 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.104+0000 7f8156b15640 1 -- 172.21.15.193:0/420832757 shutdown_connections 2024-03-25T18:03:50.105 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.104+0000 7f8156b15640 1 -- 172.21.15.193:0/420832757 >> 172.21.15.193:0/420832757 conn(0x7f81500b7f40 msgr2=0x7f8150074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:50.105 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.104+0000 7f8156b15640 1 -- 172.21.15.193:0/420832757 shutdown_connections 2024-03-25T18:03:50.105 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.104+0000 7f8156b15640 1 -- 172.21.15.193:0/420832757 wait complete. 2024-03-25T18:03:50.105 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.104+0000 7f8156b15640 1 librados: shutdown 2024-03-25T18:03:50.115 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new b3c5bd8b-e425-40d6-9208-aa973ce3c777 3 2024-03-25T18:03:50.115 INFO:tasks.ceph.mgr.y.smithi066.stderr:2024-03-25T18:03:50.115+0000 7fc1b688f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:03:50.163 INFO:tasks.ceph.mgr.y.smithi066.stderr:2024-03-25T18:03:50.163+0000 7fc1b688f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:03:50.163 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T18:03:50.164+0000 7f5e09feb040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:03:50.213 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.214+0000 7f7b74332640 10 monclient: get_monmap_and_config 2024-03-25T18:03:50.214 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.214+0000 7f7b74332640 10 monclient: build_initial_monmap 2024-03-25T18:03:50.214 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.214+0000 7f7b74332640 10 monclient: monmap: 2024-03-25T18:03:50.214 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T18:03:50.214 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:50.214 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:03:50.214562+0000 2024-03-25T18:03:50.214 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:03:50.214562+0000 2024-03-25T18:03:50.214 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:50.214 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:50.214 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:03:50.214 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:03:50.214 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:03:50.214 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:50.216 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.214+0000 7f7b74332640 1 Processor -- start 2024-03-25T18:03:50.216 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.214+0000 7f7b74332640 1 -- start start 2024-03-25T18:03:50.216 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.214+0000 7f7b74332640 10 monclient: init 2024-03-25T18:03:50.216 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.215+0000 7f7b74332640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:50.216 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.215+0000 7f7b74332640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:03:50.216 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.215+0000 7f7b74332640 10 monclient(hunting): picked mon.noname-a con 0x7f7b6c0b4700 addr v1:172.21.15.66:6789/0 2024-03-25T18:03:50.216 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.215+0000 7f7b74332640 10 monclient(hunting): picked mon.noname-c con 0x7f7b6c142a20 addr v1:172.21.15.193:6789/0 2024-03-25T18:03:50.216 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.215+0000 7f7b74332640 10 monclient(hunting): picked mon.noname-b con 0x7f7b6c0a96b0 addr v1:172.21.15.66:6790/0 2024-03-25T18:03:50.216 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.215+0000 7f7b74332640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6c0a9d60 con 0x7f7b6c0b4700 2024-03-25T18:03:50.216 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.215+0000 7f7b74332640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6c13ff10 con 0x7f7b6c142a20 2024-03-25T18:03:50.217 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.215+0000 7f7b74332640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6c0aa1f0 con 0x7f7b6c0a96b0 2024-03-25T18:03:50.217 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.215+0000 7f7b74332640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:50.217 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.215+0000 7f7b74332640 10 monclient(hunting): authenticate will time out at 6890.548451s 2024-03-25T18:03:50.217 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.216+0000 7f7b718cd640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f7b6c142a20 0x7f7b6c13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:47186/0 (socket says 172.21.15.193:47186) 2024-03-25T18:03:50.217 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.216+0000 7f7b718cd640 1 -- 172.21.15.193:0/1349413866 learned_addr learned my addr 172.21.15.193:0/1349413866 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T18:03:50.218 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.218+0000 7f7b710cc640 1 -- 172.21.15.193:0/1349413866 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 771406677 0 0) 0x7f7b6c13ff10 con 0x7f7b6c142a20 2024-03-25T18:03:50.218 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.218+0000 7f7b710cc640 10 monclient(hunting): _negotiate 2024-03-25T18:03:50.218 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.218+0000 7f7b710cc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:50.218 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.218+0000 7f7b710cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:50.218 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.218+0000 7f7b710cc640 10 monclient(hunting): my global_id is 4129 2024-03-25T18:03:50.219 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.218+0000 7f7b710cc640 1 -- 172.21.15.193:0/1349413866 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b58003520 con 0x7f7b6c142a20 2024-03-25T18:03:50.219 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.218+0000 7f7b710cc640 1 -- 172.21.15.193:0/1349413866 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 759160135 0 0) 0x7f7b6c0aa1f0 con 0x7f7b6c0a96b0 2024-03-25T18:03:50.219 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.218+0000 7f7b710cc640 10 monclient(hunting): _negotiate 2024-03-25T18:03:50.219 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.218+0000 7f7b710cc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:50.219 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.218+0000 7f7b710cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:50.219 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.218+0000 7f7b710cc640 10 monclient(hunting): my global_id is 4130 2024-03-25T18:03:50.219 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.218+0000 7f7b710cc640 1 -- 172.21.15.193:0/1349413866 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b6c13ff10 con 0x7f7b6c0a96b0 2024-03-25T18:03:50.219 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.218+0000 7f7b710cc640 1 -- 172.21.15.193:0/1349413866 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2752185417 0 0) 0x7f7b6c0a9d60 con 0x7f7b6c0b4700 2024-03-25T18:03:50.219 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.218+0000 7f7b710cc640 10 monclient(hunting): _negotiate 2024-03-25T18:03:50.219 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.218+0000 7f7b710cc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:50.219 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.218+0000 7f7b710cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:50.219 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.218+0000 7f7b710cc640 10 monclient(hunting): my global_id is 4134 2024-03-25T18:03:50.219 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.218+0000 7f7b710cc640 1 -- 172.21.15.193:0/1349413866 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b6c0aa1f0 con 0x7f7b6c0b4700 2024-03-25T18:03:50.221 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.221+0000 7f7b710cc640 1 -- 172.21.15.193:0/1349413866 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3326601212 0 0) 0x7f7b58003520 con 0x7f7b6c142a20 2024-03-25T18:03:50.221 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.221+0000 7f7b710cc640 1 -- 172.21.15.193:0/1349413866 >> v1:172.21.15.66:6790/0 conn(0x7f7b6c0a96b0 legacy=0x7f7b6c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:50.221 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.221+0000 7f7b710cc640 1 -- 172.21.15.193:0/1349413866 >> v1:172.21.15.66:6789/0 conn(0x7f7b6c0b4700 legacy=0x7f7b6c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:50.221 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.221+0000 7f7b710cc640 10 monclient: _finish_hunting 0 2024-03-25T18:03:50.222 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.221+0000 7f7b710cc640 1 monclient: found mon.noname-c 2024-03-25T18:03:50.222 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.221+0000 7f7b710cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:50.222 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.221+0000 7f7b710cc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:03:50.222 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.221+0000 7f7b710cc640 1 -- 172.21.15.193:0/1349413866 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6c0b4ae0 con 0x7f7b6c142a20 2024-03-25T18:03:50.222 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.221+0000 7f7b710cc640 10 monclient: _finish_auth 0 2024-03-25T18:03:50.222 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.221+0000 7f7b710cc640 10 monclient: _check_auth_tickets 2024-03-25T18:03:50.222 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.221+0000 7f7b710cc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:50.222 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.221+0000 7f7b74332640 5 monclient: authenticate success, global_id 4129 2024-03-25T18:03:50.222 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.221+0000 7f7b74332640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:50.222 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.222+0000 7f7b710cc640 1 -- 172.21.15.193:0/1349413866 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7b5c001650 con 0x7f7b6c142a20 2024-03-25T18:03:50.222 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.222+0000 7f7b710cc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:50.222 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.222+0000 7f7b710cc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:03:50.222 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.222+0000 7f7b710cc640 10 monclient: dump: 2024-03-25T18:03:50.222 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:50.222 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:50.223 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:50.223 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:50.223 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:50.223 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:50.223 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:50.223 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:50.223 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:50.223 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:50.223 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.222+0000 7f7b74332640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:50.226 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.223+0000 7f7b710cc640 1 -- 172.21.15.193:0/1349413866 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b5c0018a0 con 0x7f7b6c142a20 2024-03-25T18:03:50.226 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.223+0000 7f7b710cc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:50.226 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.223+0000 7f7b74332640 10 monclient: get_monmap_and_config success 2024-03-25T18:03:50.226 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.223+0000 7f7b74332640 4 set_mon_vals no callback set 2024-03-25T18:03:50.226 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.223+0000 7f7b74332640 10 monclient: shutdown 2024-03-25T18:03:50.226 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.223+0000 7f7b74332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:50.226 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.223+0000 7f7b74332640 1 -- 172.21.15.193:0/1349413866 >> v1:172.21.15.193:6789/0 conn(0x7f7b6c142a20 legacy=0x7f7b6c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:50.226 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.223+0000 7f7b74332640 1 -- 172.21.15.193:0/1349413866 shutdown_connections 2024-03-25T18:03:50.226 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.223+0000 7f7b74332640 1 -- 172.21.15.193:0/1349413866 >> 172.21.15.193:0/1349413866 conn(0x7f7b6c0668e0 msgr2=0x7f7b6c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:50.227 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.224+0000 7f7b74332640 1 -- 172.21.15.193:0/1349413866 shutdown_connections 2024-03-25T18:03:50.227 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.224+0000 7f7b74332640 1 -- 172.21.15.193:0/1349413866 wait complete. 2024-03-25T18:03:50.227 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.224+0000 7f7b74332640 10 monclient: build_initial_monmap 2024-03-25T18:03:50.227 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.224+0000 7f7b74332640 10 monclient: monmap: 2024-03-25T18:03:50.227 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T18:03:50.227 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:03:50.227 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 0.000000 2024-03-25T18:03:50.227 INFO:teuthology.orchestra.run.smithi193.stderr:created 0.000000 2024-03-25T18:03:50.227 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:50.227 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 1 2024-03-25T18:03:50.227 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:03:50.227 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:03:50.227 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:03:50.227 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:50.227 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.225+0000 7f7b74332640 1 librados: starting msgr at 2024-03-25T18:03:50.227 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.225+0000 7f7b74332640 1 librados: starting objecter 2024-03-25T18:03:50.228 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.225+0000 7f7b74332640 1 Processor -- start 2024-03-25T18:03:50.228 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.225+0000 7f7b74332640 1 -- start start 2024-03-25T18:03:50.228 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.225+0000 7f7b74332640 1 librados: setting wanted keys 2024-03-25T18:03:50.228 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.225+0000 7f7b74332640 1 librados: calling monclient init 2024-03-25T18:03:50.228 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.225+0000 7f7b74332640 10 monclient: init 2024-03-25T18:03:50.228 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.225+0000 7f7b74332640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:50.228 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.225+0000 7f7b74332640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:03:50.228 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.225+0000 7f7b74332640 10 monclient(hunting): picked mon.noname-a con 0x7f7b6c0a96b0 addr v1:172.21.15.66:6789/0 2024-03-25T18:03:50.228 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.225+0000 7f7b74332640 10 monclient(hunting): picked mon.noname-c con 0x7f7b6c0b4700 addr v1:172.21.15.66:6790/0 2024-03-25T18:03:50.228 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.225+0000 7f7b74332640 10 monclient(hunting): picked mon.noname-b con 0x7f7b6c142a20 addr v1:172.21.15.193:6789/0 2024-03-25T18:03:50.228 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.225+0000 7f7b74332640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6c13c960 con 0x7f7b6c0a96b0 2024-03-25T18:03:50.228 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.225+0000 7f7b74332640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6c13cbd0 con 0x7f7b6c142a20 2024-03-25T18:03:50.228 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.225+0000 7f7b74332640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6c0ab920 con 0x7f7b6c0b4700 2024-03-25T18:03:50.228 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.225+0000 7f7b74332640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:50.228 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.225+0000 7f7b74332640 10 monclient(hunting): authenticate will time out at 6890.558580s 2024-03-25T18:03:50.229 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.226+0000 7f7b718cd640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f7b6c0b4700 0x7f7b6c138580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.193:44896/0 (socket says 172.21.15.193:44896) 2024-03-25T18:03:50.229 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.226+0000 7f7b718cd640 1 -- 172.21.15.193:0/1123105061 learned_addr learned my addr 172.21.15.193:0/1123105061 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T18:03:50.229 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.226+0000 7f7b62ffd640 10 client.?.objecter ms_handle_connect 0x7f7b6c0b4700 2024-03-25T18:03:50.229 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.226+0000 7f7b62ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:50.230 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.227+0000 7f7b62ffd640 10 client.?.objecter ms_handle_connect 0x7f7b6c142a20 2024-03-25T18:03:50.230 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.227+0000 7f7b62ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:50.230 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.227+0000 7f7b62ffd640 10 client.?.objecter ms_handle_connect 0x7f7b6c0a96b0 2024-03-25T18:03:50.230 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.227+0000 7f7b62ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:50.230 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.229+0000 7f7b62ffd640 1 -- 172.21.15.193:0/1123105061 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3799250399 0 0) 0x7f7b6c0ab920 con 0x7f7b6c0b4700 2024-03-25T18:03:50.230 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.229+0000 7f7b62ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:03:50.230 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.229+0000 7f7b62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:50.230 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.229+0000 7f7b62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:50.230 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.229+0000 7f7b62ffd640 10 monclient(hunting): my global_id is 4133 2024-03-25T18:03:50.231 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.229+0000 7f7b62ffd640 1 -- 172.21.15.193:0/1123105061 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b50004eb0 con 0x7f7b6c0b4700 2024-03-25T18:03:50.231 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.229+0000 7f7b62ffd640 1 -- 172.21.15.193:0/1123105061 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 993467539 0 0) 0x7f7b6c13cbd0 con 0x7f7b6c142a20 2024-03-25T18:03:50.231 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.229+0000 7f7b62ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:03:50.231 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.229+0000 7f7b62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:50.231 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.229+0000 7f7b62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:50.231 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.229+0000 7f7b62ffd640 10 monclient(hunting): my global_id is 4132 2024-03-25T18:03:50.231 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.229+0000 7f7b62ffd640 1 -- 172.21.15.193:0/1123105061 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b6c0ab920 con 0x7f7b6c142a20 2024-03-25T18:03:50.231 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.231+0000 7f7b62ffd640 1 -- 172.21.15.193:0/1123105061 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 88427327 0 0) 0x7f7b6c13c960 con 0x7f7b6c0a96b0 2024-03-25T18:03:50.232 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.231+0000 7f7b62ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:03:50.232 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.231+0000 7f7b62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:50.232 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.231+0000 7f7b62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:50.232 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.231+0000 7f7b62ffd640 10 monclient(hunting): my global_id is 4137 2024-03-25T18:03:50.232 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.231+0000 7f7b62ffd640 1 -- 172.21.15.193:0/1123105061 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b6c13cbd0 con 0x7f7b6c0a96b0 2024-03-25T18:03:50.234 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.234+0000 7f7b62ffd640 1 -- 172.21.15.193:0/1123105061 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2724641146 0 0) 0x7f7b50004eb0 con 0x7f7b6c0b4700 2024-03-25T18:03:50.234 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.234+0000 7f7b62ffd640 1 -- 172.21.15.193:0/1123105061 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b6c13c960 con 0x7f7b6c0b4700 2024-03-25T18:03:50.235 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.234+0000 7f7b62ffd640 1 -- 172.21.15.193:0/1123105061 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7b5c001c70 con 0x7f7b6c0b4700 2024-03-25T18:03:50.235 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.234+0000 7f7b62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:50.235 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.234+0000 7f7b62ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:03:50.235 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.234+0000 7f7b62ffd640 10 monclient(hunting): dump: 2024-03-25T18:03:50.235 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:50.235 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:50.235 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:50.235 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:50.235 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:50.235 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:50.235 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:50.235 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:50.235 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:50.236 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:50.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.234+0000 7f7b62ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:03:50.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.235+0000 7f7b62ffd640 1 -- 172.21.15.193:0/1123105061 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 740334194 0 0) 0x7f7b6c0ab920 con 0x7f7b6c142a20 2024-03-25T18:03:50.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.235+0000 7f7b62ffd640 1 -- 172.21.15.193:0/1123105061 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b50004eb0 con 0x7f7b6c142a20 2024-03-25T18:03:50.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.235+0000 7f7b62ffd640 1 -- 172.21.15.193:0/1123105061 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7b64001cb0 con 0x7f7b6c142a20 2024-03-25T18:03:50.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.235+0000 7f7b62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:50.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.235+0000 7f7b62ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:03:50.236 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.235+0000 7f7b62ffd640 10 monclient(hunting): dump: 2024-03-25T18:03:50.236 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:50.236 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:50.236 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:50.236 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:50.236 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:50.236 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:50.237 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:50.237 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:50.237 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:50.237 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:50.237 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.236+0000 7f7b62ffd640 1 -- 172.21.15.193:0/1123105061 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3327745923 0 0) 0x7f7b6c13cbd0 con 0x7f7b6c0a96b0 2024-03-25T18:03:50.237 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.236+0000 7f7b62ffd640 1 -- 172.21.15.193:0/1123105061 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b6c0ab920 con 0x7f7b6c0a96b0 2024-03-25T18:03:50.237 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.236+0000 7f7b62ffd640 1 -- 172.21.15.193:0/1123105061 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7b68001830 con 0x7f7b6c0a96b0 2024-03-25T18:03:50.238 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.236+0000 7f7b62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:50.238 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.236+0000 7f7b62ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:03:50.238 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.236+0000 7f7b62ffd640 10 monclient(hunting): dump: 2024-03-25T18:03:50.238 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:50.238 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:50.238 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:50.238 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:50.238 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:50.238 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:50.238 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:50.238 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:50.238 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:50.238 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:50.240 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.238+0000 7f7b62ffd640 1 -- 172.21.15.193:0/1123105061 <== mon.2 v1:172.21.15.66:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3742901052 0 0) 0x7f7b6c13c960 con 0x7f7b6c0b4700 2024-03-25T18:03:50.240 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.238+0000 7f7b62ffd640 1 -- 172.21.15.193:0/1123105061 >> v1:172.21.15.193:6789/0 conn(0x7f7b6c142a20 legacy=0x7f7b6c1e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:50.240 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.238+0000 7f7b62ffd640 1 -- 172.21.15.193:0/1123105061 >> v1:172.21.15.66:6789/0 conn(0x7f7b6c0a96b0 legacy=0x7f7b6c13b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:50.240 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.238+0000 7f7b62ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:03:50.240 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.238+0000 7f7b62ffd640 1 monclient: found mon.c 2024-03-25T18:03:50.240 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.238+0000 7f7b62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:50.240 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.238+0000 7f7b62ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:03:50.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.238+0000 7f7b62ffd640 1 -- 172.21.15.193:0/1123105061 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6c0abc00 con 0x7f7b6c0b4700 2024-03-25T18:03:50.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.238+0000 7f7b62ffd640 10 monclient: _finish_auth 0 2024-03-25T18:03:50.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.238+0000 7f7b62ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:03:50.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.238+0000 7f7b62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:50.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.238+0000 7f7b74332640 5 monclient: authenticate success, global_id 4133 2024-03-25T18:03:50.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.238+0000 7f7b74332640 10 monclient: _renew_subs 2024-03-25T18:03:50.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.238+0000 7f7b74332640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:03:50.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.238+0000 7f7b74332640 1 -- 172.21.15.193:0/1123105061 --> v1:172.21.15.66:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b6c0abee0 con 0x7f7b6c0b4700 2024-03-25T18:03:50.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.238+0000 7f7b74332640 10 client.4133.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:03:50.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.238+0000 7f7b74332640 10 monclient: _renew_subs 2024-03-25T18:03:50.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.238+0000 7f7b74332640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:03:50.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.238+0000 7f7b74332640 1 -- 172.21.15.193:0/1123105061 --> v1:172.21.15.66:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b6c0ac300 con 0x7f7b6c0b4700 2024-03-25T18:03:50.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.238+0000 7f7b74332640 1 librados: init done 2024-03-25T18:03:50.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.239+0000 7f7b74332640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:50.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.239+0000 7f7b74332640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:50.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.239+0000 7f7b74332640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:03:50.242 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.239+0000 7f7b74332640 1 -- 172.21.15.193:0/1123105061 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b6c04f730 con 0x7f7b6c0b4700 2024-03-25T18:03:50.263 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.263+0000 7f7b62ffd640 1 -- 172.21.15.193:0/1123105061 <== mon.2 v1:172.21.15.66:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b5c0011f0 con 0x7f7b6c0b4700 2024-03-25T18:03:50.263 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.263+0000 7f7b62ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:50.263 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.263+0000 7f7b62ffd640 1 -- 172.21.15.193:0/1123105061 <== mon.2 v1:172.21.15.66:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7b5c004cb0 con 0x7f7b6c0b4700 2024-03-25T18:03:50.263 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.263+0000 7f7b62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:50.263 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.263+0000 7f7b62ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:03:50.263 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.263+0000 7f7b62ffd640 10 monclient: dump: 2024-03-25T18:03:50.263 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:50.263 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:50.263 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:50.263 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:50.263 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:50.263 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:50.263 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:50.263 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:50.264 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:50.264 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:50.264 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.263+0000 7f7b62ffd640 1 -- 172.21.15.193:0/1123105061 <== mon.2 v1:172.21.15.66:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7b5c005260 con 0x7f7b6c0b4700 2024-03-25T18:03:50.264 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.263+0000 7f7b708cb640 4 set_mon_vals no callback set 2024-03-25T18:03:50.264 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.263+0000 7f7b62ffd640 1 -- 172.21.15.193:0/1123105061 <== mon.2 v1:172.21.15.66:6790/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 1602097098 0 0) 0x7f7b5c006890 con 0x7f7b6c0b4700 2024-03-25T18:03:50.264 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.263+0000 7f7b62ffd640 10 client.4133.objecter ms_dispatch 0x7f7b6c000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T18:03:50.264 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.263+0000 7f7b62ffd640 3 client.4133.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T18:03:50.264 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.263+0000 7f7b62ffd640 3 client.4133.objecter handle_osd_map decoding full epoch 5 2024-03-25T18:03:50.264 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.263+0000 7f7b62ffd640 20 client.4133.objecter dump_active .. 0 homeless 2024-03-25T18:03:50.265 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T18:03:50.264+0000 7f5e09feb040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:03:50.285 INFO:tasks.ceph.mgr.y.smithi066.stderr:2024-03-25T18:03:50.285+0000 7fc1b688f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:03:50.386 INFO:tasks.ceph.mgr.y.smithi066.stderr:2024-03-25T18:03:50.386+0000 7fc1b688f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:03:50.417 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T18:03:50.418+0000 7f5e09feb040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:03:50.444 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.444+0000 7f7b62ffd640 1 -- 172.21.15.193:0/1123105061 <== mon.2 v1:172.21.15.66:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7b5c0064b0 con 0x7f7b6c0b4700 2024-03-25T18:03:50.444 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.444+0000 7f7b62ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:50.444 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.444+0000 7f7b62ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:03:50.460 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T18:03:50.460+0000 7f5e09feb040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:03:50.518 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.518+0000 7f7b74332640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b3c5bd8b-e425-40d6-9208-aa973ce3c777", "id": 3}] 2024-03-25T18:03:50.518 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.518+0000 7f7b74332640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b3c5bd8b-e425-40d6-9208-aa973ce3c777", "id": 3}] 2024-03-25T18:03:50.519 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.518+0000 7f7b74332640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:03:50.519 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.518+0000 7f7b74332640 1 -- 172.21.15.193:0/1123105061 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "osd new", "uuid": "b3c5bd8b-e425-40d6-9208-aa973ce3c777", "id": 3} v 0) v1 -- 0x7f7b6c0b7960 con 0x7f7b6c0b4700 2024-03-25T18:03:50.572 INFO:tasks.ceph.mgr.y.smithi066.stderr:2024-03-25T18:03:50.572+0000 7fc1b688f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:03:50.615 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T18:03:50.616+0000 7f5e09feb040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:03:50.619 INFO:tasks.ceph.mgr.y.smithi066.stderr:2024-03-25T18:03:50.619+0000 7fc1b688f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:03:50.655 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.655+0000 7f7b62ffd640 1 -- 172.21.15.193:0/1123105061 <== mon.2 v1:172.21.15.66:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b3c5bd8b-e425-40d6-9208-aa973ce3c777", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 2988087993 0 525046306) 0x7f7b5c006aa0 con 0x7f7b6c0b4700 2024-03-25T18:03:50.655 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.655+0000 7f7b62ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b3c5bd8b-e425-40d6-9208-aa973ce3c777", "id": 3}] 2024-03-25T18:03:50.655 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.655+0000 7f7b62ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:03:50.655 INFO:teuthology.orchestra.run.smithi193.stdout:3 2024-03-25T18:03:50.656 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.656+0000 7f7b74332640 10 librados: watch_flush enter 2024-03-25T18:03:50.656 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.656+0000 7f7b74332640 10 librados: watch_flush exit 2024-03-25T18:03:50.656 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.656+0000 7f7b74332640 20 client.4133.objecter shutdown clearing up homeless session... 2024-03-25T18:03:50.656 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.656+0000 7f7b74332640 10 client.4133.objecter successfully canceled tick 2024-03-25T18:03:50.656 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.656+0000 7f7b74332640 10 monclient: shutdown 2024-03-25T18:03:50.657 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.656+0000 7f7b74332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:50.657 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.656+0000 7f7b74332640 1 -- 172.21.15.193:0/1123105061 >> v1:172.21.15.66:6790/0 conn(0x7f7b6c0b4700 legacy=0x7f7b6c138580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:50.657 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.656+0000 7f7b74332640 1 -- 172.21.15.193:0/1123105061 shutdown_connections 2024-03-25T18:03:50.657 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.656+0000 7f7b74332640 1 -- 172.21.15.193:0/1123105061 >> 172.21.15.193:0/1123105061 conn(0x7f7b6c0668e0 msgr2=0x7f7b6c13e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:50.657 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.656+0000 7f7b74332640 1 -- 172.21.15.193:0/1123105061 shutdown_connections 2024-03-25T18:03:50.657 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.656+0000 7f7b74332640 1 -- 172.21.15.193:0/1123105061 wait complete. 2024-03-25T18:03:50.657 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.656+0000 7f7b74332640 1 librados: shutdown 2024-03-25T18:03:50.666 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new 34c9396f-b15a-43ed-aef7-ef55aa31d14d 4 2024-03-25T18:03:50.766 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.765+0000 7f9d650d3640 10 monclient: get_monmap_and_config 2024-03-25T18:03:50.766 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.765+0000 7f9d650d3640 10 monclient: build_initial_monmap 2024-03-25T18:03:50.766 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.765+0000 7f9d650d3640 10 monclient: monmap: 2024-03-25T18:03:50.766 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T18:03:50.766 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:50.766 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:03:50.766082+0000 2024-03-25T18:03:50.766 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:03:50.766082+0000 2024-03-25T18:03:50.766 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:50.766 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:50.766 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:03:50.766 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:03:50.766 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:03:50.767 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:50.767 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.766+0000 7f9d650d3640 1 Processor -- start 2024-03-25T18:03:50.767 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.766+0000 7f9d650d3640 1 -- start start 2024-03-25T18:03:50.767 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.766+0000 7f9d650d3640 10 monclient: init 2024-03-25T18:03:50.768 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.766+0000 7f9d650d3640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:50.768 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.766+0000 7f9d650d3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:03:50.768 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.766+0000 7f9d650d3640 10 monclient(hunting): picked mon.noname-a con 0x7f9d600b4700 addr v1:172.21.15.66:6789/0 2024-03-25T18:03:50.768 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.766+0000 7f9d650d3640 10 monclient(hunting): picked mon.noname-c con 0x7f9d60142a20 addr v1:172.21.15.193:6789/0 2024-03-25T18:03:50.768 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.766+0000 7f9d650d3640 10 monclient(hunting): picked mon.noname-b con 0x7f9d600a96b0 addr v1:172.21.15.66:6790/0 2024-03-25T18:03:50.768 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.767+0000 7f9d650d3640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d600a9d60 con 0x7f9d600b4700 2024-03-25T18:03:50.768 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.767+0000 7f9d650d3640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d6013ff10 con 0x7f9d60142a20 2024-03-25T18:03:50.768 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.767+0000 7f9d650d3640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d600aa1f0 con 0x7f9d600a96b0 2024-03-25T18:03:50.768 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.767+0000 7f9d650d3640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:50.768 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.767+0000 7f9d650d3640 10 monclient(hunting): authenticate will time out at 6891.099843s 2024-03-25T18:03:50.769 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.767+0000 7f9d5e59c640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f9d60142a20 0x7f9d6013f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:47216/0 (socket says 172.21.15.193:47216) 2024-03-25T18:03:50.769 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.767+0000 7f9d5e59c640 1 -- 172.21.15.193:0/2842835992 learned_addr learned my addr 172.21.15.193:0/2842835992 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T18:03:50.769 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.769+0000 7f9d5dd9b640 1 -- 172.21.15.193:0/2842835992 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308192168 0 0) 0x7f9d6013ff10 con 0x7f9d60142a20 2024-03-25T18:03:50.769 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.770+0000 7f9d5dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:03:50.769 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.770+0000 7f9d5dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:50.769 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.770+0000 7f9d5dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:50.770 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.770+0000 7f9d5dd9b640 10 monclient(hunting): my global_id is 4135 2024-03-25T18:03:50.770 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.770+0000 7f9d5dd9b640 1 -- 172.21.15.193:0/2842835992 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d4c003520 con 0x7f9d60142a20 2024-03-25T18:03:50.770 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.770+0000 7f9d5dd9b640 1 -- 172.21.15.193:0/2842835992 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 918268042 0 0) 0x7f9d600aa1f0 con 0x7f9d600a96b0 2024-03-25T18:03:50.770 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.770+0000 7f9d5dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:03:50.770 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.770+0000 7f9d5dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:50.770 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.770+0000 7f9d5dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:50.770 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.770+0000 7f9d5dd9b640 10 monclient(hunting): my global_id is 4136 2024-03-25T18:03:50.770 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.770+0000 7f9d5dd9b640 1 -- 172.21.15.193:0/2842835992 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d6013ff10 con 0x7f9d600a96b0 2024-03-25T18:03:50.771 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.771+0000 7f9d5dd9b640 1 -- 172.21.15.193:0/2842835992 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874793464 0 0) 0x7f9d600a9d60 con 0x7f9d600b4700 2024-03-25T18:03:50.771 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.771+0000 7f9d5dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:03:50.771 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.771+0000 7f9d5dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:50.771 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.771+0000 7f9d5dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:50.771 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.771+0000 7f9d5dd9b640 10 monclient(hunting): my global_id is 4140 2024-03-25T18:03:50.771 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.771+0000 7f9d5dd9b640 1 -- 172.21.15.193:0/2842835992 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d600aa1f0 con 0x7f9d600b4700 2024-03-25T18:03:50.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.773+0000 7f9d5dd9b640 1 -- 172.21.15.193:0/2842835992 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2986167006 0 0) 0x7f9d4c003520 con 0x7f9d60142a20 2024-03-25T18:03:50.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.773+0000 7f9d5dd9b640 1 -- 172.21.15.193:0/2842835992 >> v1:172.21.15.66:6790/0 conn(0x7f9d600a96b0 legacy=0x7f9d6013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:50.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.773+0000 7f9d5dd9b640 1 -- 172.21.15.193:0/2842835992 >> v1:172.21.15.66:6789/0 conn(0x7f9d600b4700 legacy=0x7f9d60142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:50.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.773+0000 7f9d5dd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:03:50.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.773+0000 7f9d5dd9b640 1 monclient: found mon.noname-c 2024-03-25T18:03:50.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.773+0000 7f9d5dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:50.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.773+0000 7f9d5dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:03:50.773 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.773+0000 7f9d5dd9b640 1 -- 172.21.15.193:0/2842835992 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d600b4ae0 con 0x7f9d60142a20 2024-03-25T18:03:50.774 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.773+0000 7f9d5dd9b640 10 monclient: _finish_auth 0 2024-03-25T18:03:50.774 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.773+0000 7f9d5dd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:03:50.774 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.773+0000 7f9d5dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:50.774 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.773+0000 7f9d5dd9b640 1 -- 172.21.15.193:0/2842835992 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f9d48001690 con 0x7f9d60142a20 2024-03-25T18:03:50.774 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.773+0000 7f9d650d3640 5 monclient: authenticate success, global_id 4135 2024-03-25T18:03:50.774 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.773+0000 7f9d650d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:50.774 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.773+0000 7f9d5dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:50.774 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.773+0000 7f9d5dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:03:50.774 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.773+0000 7f9d5dd9b640 10 monclient: dump: 2024-03-25T18:03:50.774 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:50.774 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:50.774 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:50.774 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:50.774 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:50.775 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:50.775 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:50.775 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:50.775 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:50.775 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:50.775 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.773+0000 7f9d650d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:50.778 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.774+0000 7f9d5dd9b640 1 -- 172.21.15.193:0/2842835992 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d480017f0 con 0x7f9d60142a20 2024-03-25T18:03:50.778 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.774+0000 7f9d5dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:50.778 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.774+0000 7f9d5dd9b640 1 -- 172.21.15.193:0/2842835992 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f9d48001c00 con 0x7f9d60142a20 2024-03-25T18:03:50.778 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.774+0000 7f9d650d3640 10 monclient: get_monmap_and_config success 2024-03-25T18:03:50.778 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.774+0000 7f9d650d3640 4 set_mon_vals no callback set 2024-03-25T18:03:50.778 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.774+0000 7f9d5dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:50.778 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.774+0000 7f9d5dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:03:50.779 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.774+0000 7f9d5dd9b640 10 monclient: dump: 2024-03-25T18:03:50.779 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:50.779 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:50.779 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:50.779 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:50.779 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:50.779 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:50.779 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:50.779 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:50.779 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:50.779 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:50.779 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.775+0000 7f9d650d3640 10 monclient: shutdown 2024-03-25T18:03:50.779 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.775+0000 7f9d650d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:50.779 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.775+0000 7f9d650d3640 1 -- 172.21.15.193:0/2842835992 >> v1:172.21.15.193:6789/0 conn(0x7f9d60142a20 legacy=0x7f9d6013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:50.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.775+0000 7f9d650d3640 1 -- 172.21.15.193:0/2842835992 shutdown_connections 2024-03-25T18:03:50.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.775+0000 7f9d650d3640 1 -- 172.21.15.193:0/2842835992 >> 172.21.15.193:0/2842835992 conn(0x7f9d600668e0 msgr2=0x7f9d60066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:50.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.775+0000 7f9d650d3640 1 -- 172.21.15.193:0/2842835992 shutdown_connections 2024-03-25T18:03:50.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.775+0000 7f9d650d3640 1 -- 172.21.15.193:0/2842835992 wait complete. 2024-03-25T18:03:50.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.776+0000 7f9d650d3640 10 monclient: build_initial_monmap 2024-03-25T18:03:50.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.776+0000 7f9d650d3640 10 monclient: monmap: 2024-03-25T18:03:50.780 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T18:03:50.780 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:03:50.780 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 0.000000 2024-03-25T18:03:50.780 INFO:teuthology.orchestra.run.smithi193.stderr:created 0.000000 2024-03-25T18:03:50.780 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:50.780 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 1 2024-03-25T18:03:50.780 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:03:50.780 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:03:50.780 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:03:50.781 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:50.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.776+0000 7f9d650d3640 1 librados: starting msgr at 2024-03-25T18:03:50.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.776+0000 7f9d650d3640 1 librados: starting objecter 2024-03-25T18:03:50.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.777+0000 7f9d650d3640 1 Processor -- start 2024-03-25T18:03:50.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.777+0000 7f9d650d3640 1 -- start start 2024-03-25T18:03:50.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.777+0000 7f9d650d3640 1 librados: setting wanted keys 2024-03-25T18:03:50.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.777+0000 7f9d650d3640 1 librados: calling monclient init 2024-03-25T18:03:50.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.777+0000 7f9d650d3640 10 monclient: init 2024-03-25T18:03:50.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.777+0000 7f9d650d3640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:50.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.777+0000 7f9d650d3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:03:50.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.777+0000 7f9d650d3640 10 monclient(hunting): picked mon.noname-c con 0x7f9d600a96b0 addr v1:172.21.15.66:6790/0 2024-03-25T18:03:50.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.777+0000 7f9d650d3640 10 monclient(hunting): picked mon.noname-a con 0x7f9d600b4700 addr v1:172.21.15.66:6789/0 2024-03-25T18:03:50.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.777+0000 7f9d650d3640 10 monclient(hunting): picked mon.noname-b con 0x7f9d60142a20 addr v1:172.21.15.193:6789/0 2024-03-25T18:03:50.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.777+0000 7f9d650d3640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d601dabd0 con 0x7f9d600b4700 2024-03-25T18:03:50.781 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.777+0000 7f9d650d3640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d601d3fd0 con 0x7f9d60142a20 2024-03-25T18:03:50.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.777+0000 7f9d650d3640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d601d4160 con 0x7f9d600a96b0 2024-03-25T18:03:50.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.777+0000 7f9d650d3640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:50.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.777+0000 7f9d650d3640 10 monclient(hunting): authenticate will time out at 6891.110603s 2024-03-25T18:03:50.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.778+0000 7f9d5f59e640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f9d60142a20 0x7f9d60137d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:47232/0 (socket says 172.21.15.193:47232) 2024-03-25T18:03:50.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.778+0000 7f9d5f59e640 1 -- 172.21.15.193:0/748864210 learned_addr learned my addr 172.21.15.193:0/748864210 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T18:03:50.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.778+0000 7f9d577fe640 10 client.?.objecter ms_handle_connect 0x7f9d60142a20 2024-03-25T18:03:50.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.778+0000 7f9d577fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:50.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.778+0000 7f9d577fe640 10 client.?.objecter ms_handle_connect 0x7f9d600a96b0 2024-03-25T18:03:50.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.778+0000 7f9d577fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:50.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.779+0000 7f9d577fe640 10 client.?.objecter ms_handle_connect 0x7f9d600b4700 2024-03-25T18:03:50.782 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.779+0000 7f9d577fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:50.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.780+0000 7f9d577fe640 1 -- 172.21.15.193:0/748864210 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2319111646 0 0) 0x7f9d601d4160 con 0x7f9d600a96b0 2024-03-25T18:03:50.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.780+0000 7f9d577fe640 10 monclient(hunting): _negotiate 2024-03-25T18:03:50.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.780+0000 7f9d577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:50.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.780+0000 7f9d577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:50.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.780+0000 7f9d577fe640 10 monclient(hunting): my global_id is 4139 2024-03-25T18:03:50.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.780+0000 7f9d577fe640 1 -- 172.21.15.193:0/748864210 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d40004eb0 con 0x7f9d600a96b0 2024-03-25T18:03:50.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.781+0000 7f9d577fe640 1 -- 172.21.15.193:0/748864210 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1865005475 0 0) 0x7f9d601dabd0 con 0x7f9d600b4700 2024-03-25T18:03:50.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.781+0000 7f9d577fe640 10 monclient(hunting): _negotiate 2024-03-25T18:03:50.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.781+0000 7f9d577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:50.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.781+0000 7f9d577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:50.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.781+0000 7f9d577fe640 10 monclient(hunting): my global_id is 4143 2024-03-25T18:03:50.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.781+0000 7f9d577fe640 1 -- 172.21.15.193:0/748864210 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d601d4160 con 0x7f9d600b4700 2024-03-25T18:03:50.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.782+0000 7f9d577fe640 1 -- 172.21.15.193:0/748864210 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689466068 0 0) 0x7f9d601d3fd0 con 0x7f9d60142a20 2024-03-25T18:03:50.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.782+0000 7f9d577fe640 10 monclient(hunting): _negotiate 2024-03-25T18:03:50.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.782+0000 7f9d577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:50.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.782+0000 7f9d577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:50.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.782+0000 7f9d577fe640 10 monclient(hunting): my global_id is 4138 2024-03-25T18:03:50.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.782+0000 7f9d577fe640 1 -- 172.21.15.193:0/748864210 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d601dabd0 con 0x7f9d60142a20 2024-03-25T18:03:50.786 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.786+0000 7f9d577fe640 1 -- 172.21.15.193:0/748864210 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 773584012 0 0) 0x7f9d40004eb0 con 0x7f9d600a96b0 2024-03-25T18:03:50.786 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.786+0000 7f9d577fe640 1 -- 172.21.15.193:0/748864210 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d601d3fd0 con 0x7f9d600a96b0 2024-03-25T18:03:50.786 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.786+0000 7f9d577fe640 1 -- 172.21.15.193:0/748864210 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f9d500018f0 con 0x7f9d600a96b0 2024-03-25T18:03:50.786 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.786+0000 7f9d577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:50.786 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.786+0000 7f9d577fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:03:50.786 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.786+0000 7f9d577fe640 10 monclient(hunting): dump: 2024-03-25T18:03:50.786 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:50.786 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:50.786 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:50.786 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:50.786 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:50.786 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:50.786 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:50.787 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:50.787 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:50.787 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:50.787 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.786+0000 7f9d577fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:03:50.788 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.786+0000 7f9d577fe640 1 -- 172.21.15.193:0/748864210 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4136075520 0 0) 0x7f9d601d4160 con 0x7f9d600b4700 2024-03-25T18:03:50.788 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.786+0000 7f9d577fe640 1 -- 172.21.15.193:0/748864210 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d40004eb0 con 0x7f9d600b4700 2024-03-25T18:03:50.788 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.786+0000 7f9d577fe640 1 -- 172.21.15.193:0/748864210 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f9d48001b00 con 0x7f9d600b4700 2024-03-25T18:03:50.788 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.786+0000 7f9d577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:50.788 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.786+0000 7f9d577fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:03:50.788 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.786+0000 7f9d577fe640 10 monclient(hunting): dump: 2024-03-25T18:03:50.788 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:50.788 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:50.788 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:50.788 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:50.788 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:50.789 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:50.789 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:50.789 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:50.789 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:50.789 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:50.789 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.787+0000 7f9d577fe640 1 -- 172.21.15.193:0/748864210 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3082019679 0 0) 0x7f9d601dabd0 con 0x7f9d60142a20 2024-03-25T18:03:50.789 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.787+0000 7f9d577fe640 1 -- 172.21.15.193:0/748864210 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d601d4160 con 0x7f9d60142a20 2024-03-25T18:03:50.789 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.787+0000 7f9d577fe640 1 -- 172.21.15.193:0/748864210 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f9d58001c30 con 0x7f9d60142a20 2024-03-25T18:03:50.789 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.787+0000 7f9d577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:50.789 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.787+0000 7f9d577fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:03:50.789 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.787+0000 7f9d577fe640 10 monclient(hunting): dump: 2024-03-25T18:03:50.789 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:50.789 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:50.789 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:50.789 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:50.790 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:50.790 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:50.790 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:50.790 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:50.790 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:50.790 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:50.790 INFO:tasks.ceph.mgr.y.smithi066.stderr:2024-03-25T18:03:50.788+0000 7fc1b688f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:03:50.791 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.789+0000 7f9d577fe640 1 -- 172.21.15.193:0/748864210 <== mon.2 v1:172.21.15.66:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3884029227 0 0) 0x7f9d601d3fd0 con 0x7f9d600a96b0 2024-03-25T18:03:50.791 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.789+0000 7f9d577fe640 1 -- 172.21.15.193:0/748864210 >> v1:172.21.15.193:6789/0 conn(0x7f9d60142a20 legacy=0x7f9d60137d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:50.791 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.789+0000 7f9d577fe640 1 -- 172.21.15.193:0/748864210 >> v1:172.21.15.66:6789/0 conn(0x7f9d600b4700 legacy=0x7f9d601d4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:50.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.789+0000 7f9d577fe640 10 monclient: _finish_hunting 0 2024-03-25T18:03:50.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.789+0000 7f9d577fe640 1 monclient: found mon.c 2024-03-25T18:03:50.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.789+0000 7f9d577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:50.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.789+0000 7f9d577fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:03:50.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.789+0000 7f9d577fe640 1 -- 172.21.15.193:0/748864210 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d601db770 con 0x7f9d600a96b0 2024-03-25T18:03:50.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.790+0000 7f9d577fe640 10 monclient: _finish_auth 0 2024-03-25T18:03:50.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.790+0000 7f9d577fe640 10 monclient: _check_auth_tickets 2024-03-25T18:03:50.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.790+0000 7f9d577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:50.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.790+0000 7f9d650d3640 5 monclient: authenticate success, global_id 4139 2024-03-25T18:03:50.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.790+0000 7f9d650d3640 10 monclient: _renew_subs 2024-03-25T18:03:50.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.790+0000 7f9d650d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:03:50.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.790+0000 7f9d650d3640 1 -- 172.21.15.193:0/748864210 --> v1:172.21.15.66:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d601367c0 con 0x7f9d600a96b0 2024-03-25T18:03:50.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.790+0000 7f9d650d3640 10 client.4139.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:03:50.792 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.790+0000 7f9d650d3640 10 monclient: _renew_subs 2024-03-25T18:03:50.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.790+0000 7f9d650d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:03:50.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.790+0000 7f9d650d3640 1 -- 172.21.15.193:0/748864210 --> v1:172.21.15.66:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d601d4f00 con 0x7f9d600a96b0 2024-03-25T18:03:50.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.790+0000 7f9d650d3640 1 librados: init done 2024-03-25T18:03:50.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.790+0000 7f9d650d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:50.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.790+0000 7f9d650d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:50.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.790+0000 7f9d650d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:03:50.793 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.790+0000 7f9d650d3640 1 -- 172.21.15.193:0/748864210 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d6004f730 con 0x7f9d600a96b0 2024-03-25T18:03:50.892 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.893+0000 7f9d577fe640 1 -- 172.21.15.193:0/748864210 <== mon.2 v1:172.21.15.66:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d50003610 con 0x7f9d600a96b0 2024-03-25T18:03:50.893 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.893+0000 7f9d577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:50.893 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.893+0000 7f9d577fe640 1 -- 172.21.15.193:0/748864210 <== mon.2 v1:172.21.15.66:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f9d50003920 con 0x7f9d600a96b0 2024-03-25T18:03:50.893 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.893+0000 7f9d577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:50.893 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.893+0000 7f9d577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:03:50.893 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.893+0000 7f9d577fe640 10 monclient: dump: 2024-03-25T18:03:50.893 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:50.893 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:50.893 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:50.893 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:50.893 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:50.893 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:50.893 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:50.893 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:50.894 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:50.894 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:50.894 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.893+0000 7f9d577fe640 1 -- 172.21.15.193:0/748864210 <== mon.2 v1:172.21.15.66:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9d50003e50 con 0x7f9d600a96b0 2024-03-25T18:03:50.894 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.893+0000 7f9d5d59a640 4 set_mon_vals no callback set 2024-03-25T18:03:50.894 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.894+0000 7f9d577fe640 1 -- 172.21.15.193:0/748864210 <== mon.2 v1:172.21.15.66:6790/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 512946622 0 0) 0x7f9d50005520 con 0x7f9d600a96b0 2024-03-25T18:03:50.894 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.894+0000 7f9d577fe640 10 client.4139.objecter ms_dispatch 0x7f9d60000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T18:03:50.894 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.894+0000 7f9d577fe640 3 client.4139.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T18:03:50.895 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.894+0000 7f9d577fe640 3 client.4139.objecter handle_osd_map decoding full epoch 6 2024-03-25T18:03:50.895 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.894+0000 7f9d577fe640 20 client.4139.objecter dump_active .. 0 homeless 2024-03-25T18:03:50.895 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.894+0000 7f9d577fe640 1 -- 172.21.15.193:0/748864210 <== mon.2 v1:172.21.15.66:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9d50005010 con 0x7f9d600a96b0 2024-03-25T18:03:50.895 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.894+0000 7f9d577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:50.895 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.894+0000 7f9d577fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:03:50.965 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.965+0000 7f9d650d3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "34c9396f-b15a-43ed-aef7-ef55aa31d14d", "id": 4}] 2024-03-25T18:03:50.965 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.965+0000 7f9d650d3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "34c9396f-b15a-43ed-aef7-ef55aa31d14d", "id": 4}] 2024-03-25T18:03:50.965 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.965+0000 7f9d650d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:03:50.965 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:50.965+0000 7f9d650d3640 1 -- 172.21.15.193:0/748864210 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "osd new", "uuid": "34c9396f-b15a-43ed-aef7-ef55aa31d14d", "id": 4} v 0) v1 -- 0x7f9d600b2510 con 0x7f9d600a96b0 2024-03-25T18:03:51.035 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.035+0000 7f9d577fe640 1 -- 172.21.15.193:0/748864210 <== mon.2 v1:172.21.15.66:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "34c9396f-b15a-43ed-aef7-ef55aa31d14d", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 3904221080 0 1697012583) 0x7f9d50005300 con 0x7f9d600a96b0 2024-03-25T18:03:51.035 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.035+0000 7f9d577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "34c9396f-b15a-43ed-aef7-ef55aa31d14d", "id": 4}] 2024-03-25T18:03:51.035 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.035+0000 7f9d577fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:03:51.035 INFO:teuthology.orchestra.run.smithi193.stdout:4 2024-03-25T18:03:51.037 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.036+0000 7f9d650d3640 10 librados: watch_flush enter 2024-03-25T18:03:51.037 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.036+0000 7f9d650d3640 10 librados: watch_flush exit 2024-03-25T18:03:51.037 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.036+0000 7f9d650d3640 20 client.4139.objecter shutdown clearing up homeless session... 2024-03-25T18:03:51.037 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.036+0000 7f9d650d3640 10 client.4139.objecter successfully canceled tick 2024-03-25T18:03:51.037 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.036+0000 7f9d650d3640 10 monclient: shutdown 2024-03-25T18:03:51.037 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.036+0000 7f9d650d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:51.037 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.036+0000 7f9d650d3640 1 -- 172.21.15.193:0/748864210 >> v1:172.21.15.66:6790/0 conn(0x7f9d600a96b0 legacy=0x7f9d601db060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:51.037 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.036+0000 7f9d650d3640 1 -- 172.21.15.193:0/748864210 shutdown_connections 2024-03-25T18:03:51.037 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.036+0000 7f9d650d3640 1 -- 172.21.15.193:0/748864210 >> 172.21.15.193:0/748864210 conn(0x7f9d600668e0 msgr2=0x7f9d60066cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:51.037 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.036+0000 7f9d650d3640 1 -- 172.21.15.193:0/748864210 shutdown_connections 2024-03-25T18:03:51.038 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.036+0000 7f9d650d3640 1 -- 172.21.15.193:0/748864210 wait complete. 2024-03-25T18:03:51.038 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.036+0000 7f9d650d3640 1 librados: shutdown 2024-03-25T18:03:51.046 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new 0ae5db93-018c-48ff-a423-7061d004c4c1 5 2024-03-25T18:03:51.145 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.145+0000 7fce6792b640 10 monclient: get_monmap_and_config 2024-03-25T18:03:51.146 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.145+0000 7fce6792b640 10 monclient: build_initial_monmap 2024-03-25T18:03:51.146 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.145+0000 7fce6792b640 10 monclient: monmap: 2024-03-25T18:03:51.146 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T18:03:51.146 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:51.146 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:03:51.146041+0000 2024-03-25T18:03:51.146 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:03:51.146041+0000 2024-03-25T18:03:51.146 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:51.146 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:51.146 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:03:51.146 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:03:51.146 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:03:51.147 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:51.147 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.146+0000 7fce6792b640 1 Processor -- start 2024-03-25T18:03:51.147 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.146+0000 7fce6792b640 1 -- start start 2024-03-25T18:03:51.147 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.146+0000 7fce6792b640 10 monclient: init 2024-03-25T18:03:51.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.147+0000 7fce6792b640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:51.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.147+0000 7fce6792b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:03:51.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.147+0000 7fce6792b640 10 monclient(hunting): picked mon.noname-a con 0x7fce6006c2d0 addr v1:172.21.15.66:6789/0 2024-03-25T18:03:51.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.147+0000 7fce6792b640 10 monclient(hunting): picked mon.noname-b con 0x7fce6006c6b0 addr v1:172.21.15.66:6790/0 2024-03-25T18:03:51.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.147+0000 7fce6792b640 10 monclient(hunting): picked mon.noname-c con 0x7fce60142d90 addr v1:172.21.15.193:6789/0 2024-03-25T18:03:51.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.147+0000 7fce6792b640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce600b0840 con 0x7fce6006c2d0 2024-03-25T18:03:51.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.147+0000 7fce6792b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce60143200 con 0x7fce60142d90 2024-03-25T18:03:51.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.147+0000 7fce6792b640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce6006d810 con 0x7fce6006c6b0 2024-03-25T18:03:51.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.147+0000 7fce6792b640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:51.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.147+0000 7fce6792b640 10 monclient(hunting): authenticate will time out at 6891.479925s 2024-03-25T18:03:51.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.147+0000 7fce65ec8640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fce60142d90 0x7fce6014dbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:47244/0 (socket says 172.21.15.193:47244) 2024-03-25T18:03:51.148 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.147+0000 7fce65ec8640 1 -- 172.21.15.193:0/2411872247 learned_addr learned my addr 172.21.15.193:0/2411872247 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T18:03:51.149 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.149+0000 7fce57fff640 1 -- 172.21.15.193:0/2411872247 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424990702 0 0) 0x7fce58001490 con 0x7fce60142d90 2024-03-25T18:03:51.149 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.149+0000 7fce57fff640 10 monclient(hunting): _negotiate 2024-03-25T18:03:51.150 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.149+0000 7fce57fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:51.150 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.149+0000 7fce57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:51.150 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.149+0000 7fce57fff640 10 monclient(hunting): my global_id is 4141 2024-03-25T18:03:51.150 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.150+0000 7fce57fff640 1 -- 172.21.15.193:0/2411872247 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce4c003520 con 0x7fce60142d90 2024-03-25T18:03:51.150 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.150+0000 7fce57fff640 1 -- 172.21.15.193:0/2411872247 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915303522 0 0) 0x7fce6006d810 con 0x7fce6006c6b0 2024-03-25T18:03:51.150 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.150+0000 7fce57fff640 10 monclient(hunting): _negotiate 2024-03-25T18:03:51.150 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.150+0000 7fce57fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:51.150 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.150+0000 7fce57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:51.150 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.150+0000 7fce57fff640 10 monclient(hunting): my global_id is 4142 2024-03-25T18:03:51.150 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.150+0000 7fce57fff640 1 -- 172.21.15.193:0/2411872247 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce58001490 con 0x7fce6006c6b0 2024-03-25T18:03:51.151 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.150+0000 7fce57fff640 1 -- 172.21.15.193:0/2411872247 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2666186501 0 0) 0x7fce600b0840 con 0x7fce6006c2d0 2024-03-25T18:03:51.151 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.150+0000 7fce57fff640 10 monclient(hunting): _negotiate 2024-03-25T18:03:51.151 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.150+0000 7fce57fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:51.151 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.150+0000 7fce57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:51.151 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.150+0000 7fce57fff640 10 monclient(hunting): my global_id is 4146 2024-03-25T18:03:51.151 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.150+0000 7fce57fff640 1 -- 172.21.15.193:0/2411872247 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce6006d810 con 0x7fce6006c2d0 2024-03-25T18:03:51.152 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.152+0000 7fce57fff640 1 -- 172.21.15.193:0/2411872247 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1319127575 0 0) 0x7fce4c003520 con 0x7fce60142d90 2024-03-25T18:03:51.152 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.152+0000 7fce57fff640 1 -- 172.21.15.193:0/2411872247 >> v1:172.21.15.66:6790/0 conn(0x7fce6006c6b0 legacy=0x7fce60142680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:51.152 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.152+0000 7fce57fff640 1 -- 172.21.15.193:0/2411872247 >> v1:172.21.15.66:6789/0 conn(0x7fce6006c2d0 legacy=0x7fce600afde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:51.153 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.153+0000 7fce57fff640 10 monclient: _finish_hunting 0 2024-03-25T18:03:51.153 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.153+0000 7fce57fff640 1 monclient: found mon.noname-c 2024-03-25T18:03:51.153 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.153+0000 7fce57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:51.153 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.153+0000 7fce57fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:03:51.153 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.153+0000 7fce57fff640 1 -- 172.21.15.193:0/2411872247 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce600ae260 con 0x7fce60142d90 2024-03-25T18:03:51.153 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.153+0000 7fce57fff640 10 monclient: _finish_auth 0 2024-03-25T18:03:51.153 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.153+0000 7fce57fff640 10 monclient: _check_auth_tickets 2024-03-25T18:03:51.153 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.153+0000 7fce57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:51.153 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.153+0000 7fce6792b640 5 monclient: authenticate success, global_id 4141 2024-03-25T18:03:51.153 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.153+0000 7fce6792b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:51.154 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.153+0000 7fce57fff640 1 -- 172.21.15.193:0/2411872247 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fce580019c0 con 0x7fce60142d90 2024-03-25T18:03:51.154 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.153+0000 7fce57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:51.154 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.153+0000 7fce57fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:03:51.154 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.153+0000 7fce57fff640 10 monclient: dump: 2024-03-25T18:03:51.154 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:51.154 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:51.154 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.154 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.154 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:51.155 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:51.155 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:51.155 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:51.155 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:51.155 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:51.155 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.153+0000 7fce6792b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:51.155 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.154+0000 7fce57fff640 1 -- 172.21.15.193:0/2411872247 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce58001ba0 con 0x7fce60142d90 2024-03-25T18:03:51.155 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.154+0000 7fce57fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:51.155 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.154+0000 7fce57fff640 1 -- 172.21.15.193:0/2411872247 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fce58001f30 con 0x7fce60142d90 2024-03-25T18:03:51.155 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.154+0000 7fce57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:51.155 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.154+0000 7fce57fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:03:51.155 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.154+0000 7fce57fff640 10 monclient: dump: 2024-03-25T18:03:51.155 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:51.155 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:51.155 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.155 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.156 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:51.156 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:51.156 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:51.156 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:51.156 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:51.156 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:51.156 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.154+0000 7fce6792b640 10 monclient: get_monmap_and_config success 2024-03-25T18:03:51.156 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.154+0000 7fce6792b640 4 set_mon_vals no callback set 2024-03-25T18:03:51.158 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.155+0000 7fce6792b640 10 monclient: shutdown 2024-03-25T18:03:51.159 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.155+0000 7fce6792b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:51.159 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.155+0000 7fce6792b640 1 -- 172.21.15.193:0/2411872247 >> v1:172.21.15.193:6789/0 conn(0x7fce60142d90 legacy=0x7fce6014dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:51.159 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.155+0000 7fce6792b640 1 -- 172.21.15.193:0/2411872247 shutdown_connections 2024-03-25T18:03:51.159 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.155+0000 7fce6792b640 1 -- 172.21.15.193:0/2411872247 >> 172.21.15.193:0/2411872247 conn(0x7fce600b5e40 msgr2=0x7fce600b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:51.159 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.155+0000 7fce6792b640 1 -- 172.21.15.193:0/2411872247 shutdown_connections 2024-03-25T18:03:51.159 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.155+0000 7fce6792b640 1 -- 172.21.15.193:0/2411872247 wait complete. 2024-03-25T18:03:51.159 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.156+0000 7fce6792b640 10 monclient: build_initial_monmap 2024-03-25T18:03:51.159 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.156+0000 7fce6792b640 10 monclient: monmap: 2024-03-25T18:03:51.159 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T18:03:51.159 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:03:51.159 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 0.000000 2024-03-25T18:03:51.159 INFO:teuthology.orchestra.run.smithi193.stderr:created 0.000000 2024-03-25T18:03:51.159 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:51.159 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 1 2024-03-25T18:03:51.160 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:03:51.160 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:03:51.160 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:03:51.160 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:51.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.156+0000 7fce6792b640 1 librados: starting msgr at 2024-03-25T18:03:51.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.156+0000 7fce6792b640 1 librados: starting objecter 2024-03-25T18:03:51.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.156+0000 7fce6792b640 1 Processor -- start 2024-03-25T18:03:51.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.156+0000 7fce6792b640 1 -- start start 2024-03-25T18:03:51.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.156+0000 7fce6792b640 1 librados: setting wanted keys 2024-03-25T18:03:51.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.156+0000 7fce6792b640 1 librados: calling monclient init 2024-03-25T18:03:51.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.156+0000 7fce6792b640 10 monclient: init 2024-03-25T18:03:51.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.157+0000 7fce6792b640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:51.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.157+0000 7fce6792b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:03:51.160 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.157+0000 7fce6792b640 10 monclient(hunting): picked mon.noname-a con 0x7fce6006c2d0 addr v1:172.21.15.66:6789/0 2024-03-25T18:03:51.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.157+0000 7fce6792b640 10 monclient(hunting): picked mon.noname-c con 0x7fce6006c6b0 addr v1:172.21.15.66:6790/0 2024-03-25T18:03:51.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.157+0000 7fce6792b640 10 monclient(hunting): picked mon.noname-b con 0x7fce60142d90 addr v1:172.21.15.193:6789/0 2024-03-25T18:03:51.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.157+0000 7fce6792b640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce6013ba20 con 0x7fce6006c2d0 2024-03-25T18:03:51.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.157+0000 7fce6792b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce6013bd00 con 0x7fce60142d90 2024-03-25T18:03:51.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.157+0000 7fce6792b640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce6013bfe0 con 0x7fce6006c6b0 2024-03-25T18:03:51.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.157+0000 7fce6792b640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:51.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.157+0000 7fce6792b640 10 monclient(hunting): authenticate will time out at 6891.489945s 2024-03-25T18:03:51.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.157+0000 7fce65ec8640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fce60142d90 0x7fce601dae30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:47250/0 (socket says 172.21.15.193:47250) 2024-03-25T18:03:51.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.157+0000 7fce65ec8640 1 -- 172.21.15.193:0/2219988032 learned_addr learned my addr 172.21.15.193:0/2219988032 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T18:03:51.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.157+0000 7fce55ffb640 10 client.?.objecter ms_handle_connect 0x7fce60142d90 2024-03-25T18:03:51.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.157+0000 7fce55ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:51.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.157+0000 7fce55ffb640 10 client.?.objecter ms_handle_connect 0x7fce6006c6b0 2024-03-25T18:03:51.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.157+0000 7fce55ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:51.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.158+0000 7fce55ffb640 10 client.?.objecter ms_handle_connect 0x7fce6006c2d0 2024-03-25T18:03:51.161 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.158+0000 7fce55ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:51.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.159+0000 7fce55ffb640 1 -- 172.21.15.193:0/2219988032 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3805872788 0 0) 0x7fce6013bd00 con 0x7fce60142d90 2024-03-25T18:03:51.162 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.159+0000 7fce55ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:03:51.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.159+0000 7fce55ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:51.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.159+0000 7fce55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:51.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.159+0000 7fce55ffb640 10 monclient(hunting): my global_id is 4144 2024-03-25T18:03:51.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.159+0000 7fce55ffb640 1 -- 172.21.15.193:0/2219988032 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce44004eb0 con 0x7fce60142d90 2024-03-25T18:03:51.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.160+0000 7fce55ffb640 1 -- 172.21.15.193:0/2219988032 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169521778 0 0) 0x7fce6013bfe0 con 0x7fce6006c6b0 2024-03-25T18:03:51.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.160+0000 7fce55ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:03:51.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.160+0000 7fce55ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:51.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.160+0000 7fce55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:51.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.160+0000 7fce55ffb640 10 monclient(hunting): my global_id is 4145 2024-03-25T18:03:51.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.160+0000 7fce55ffb640 1 -- 172.21.15.193:0/2219988032 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce6013bd00 con 0x7fce6006c6b0 2024-03-25T18:03:51.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.160+0000 7fce55ffb640 1 -- 172.21.15.193:0/2219988032 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4132135282 0 0) 0x7fce6013ba20 con 0x7fce6006c2d0 2024-03-25T18:03:51.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.160+0000 7fce55ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:03:51.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.160+0000 7fce55ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:51.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.160+0000 7fce55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:51.163 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.160+0000 7fce55ffb640 10 monclient(hunting): my global_id is 4149 2024-03-25T18:03:51.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.160+0000 7fce55ffb640 1 -- 172.21.15.193:0/2219988032 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce6013bfe0 con 0x7fce6006c2d0 2024-03-25T18:03:51.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.164+0000 7fce55ffb640 1 -- 172.21.15.193:0/2219988032 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3280956426 0 0) 0x7fce44004eb0 con 0x7fce60142d90 2024-03-25T18:03:51.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.164+0000 7fce55ffb640 1 -- 172.21.15.193:0/2219988032 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce6013ba20 con 0x7fce60142d90 2024-03-25T18:03:51.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.164+0000 7fce55ffb640 1 -- 172.21.15.193:0/2219988032 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fce58004b60 con 0x7fce60142d90 2024-03-25T18:03:51.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.164+0000 7fce55ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:51.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.164+0000 7fce55ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:03:51.164 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.164+0000 7fce55ffb640 10 monclient(hunting): dump: 2024-03-25T18:03:51.164 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:51.165 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:51.165 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.165 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.165 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:51.165 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:51.165 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:51.165 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:51.165 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:51.165 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:51.165 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.164+0000 7fce55ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:03:51.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.165+0000 7fce55ffb640 1 -- 172.21.15.193:0/2219988032 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 554995094 0 0) 0x7fce6013bd00 con 0x7fce6006c6b0 2024-03-25T18:03:51.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.165+0000 7fce55ffb640 1 -- 172.21.15.193:0/2219988032 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce44004eb0 con 0x7fce6006c6b0 2024-03-25T18:03:51.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.165+0000 7fce55ffb640 1 -- 172.21.15.193:0/2219988032 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fce5c001930 con 0x7fce6006c6b0 2024-03-25T18:03:51.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.165+0000 7fce55ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:51.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.165+0000 7fce55ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:03:51.166 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.165+0000 7fce55ffb640 10 monclient(hunting): dump: 2024-03-25T18:03:51.166 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:51.166 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:51.167 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.167 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.167 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:51.167 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:51.167 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:51.167 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:51.167 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:51.167 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:51.167 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.166+0000 7fce55ffb640 1 -- 172.21.15.193:0/2219988032 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4054133723 0 0) 0x7fce6013bfe0 con 0x7fce6006c2d0 2024-03-25T18:03:51.167 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.166+0000 7fce55ffb640 1 -- 172.21.15.193:0/2219988032 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce6013bd00 con 0x7fce6006c2d0 2024-03-25T18:03:51.167 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.166+0000 7fce55ffb640 1 -- 172.21.15.193:0/2219988032 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fce50001790 con 0x7fce6006c2d0 2024-03-25T18:03:51.167 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.166+0000 7fce55ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:51.167 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.166+0000 7fce55ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:03:51.167 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.166+0000 7fce55ffb640 10 monclient(hunting): dump: 2024-03-25T18:03:51.167 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:51.168 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:51.168 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.168 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.168 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:51.168 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:51.168 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:51.168 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:51.168 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:51.168 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:51.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.168+0000 7fce55ffb640 1 -- 172.21.15.193:0/2219988032 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 413407473 0 0) 0x7fce6013ba20 con 0x7fce60142d90 2024-03-25T18:03:51.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.168+0000 7fce55ffb640 1 -- 172.21.15.193:0/2219988032 >> v1:172.21.15.66:6790/0 conn(0x7fce6006c6b0 legacy=0x7fce601da720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:51.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.168+0000 7fce55ffb640 1 -- 172.21.15.193:0/2219988032 >> v1:172.21.15.66:6789/0 conn(0x7fce6006c2d0 legacy=0x7fce60137070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:51.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.168+0000 7fce55ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:03:51.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.168+0000 7fce55ffb640 1 monclient: found mon.b 2024-03-25T18:03:51.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.168+0000 7fce55ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:51.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.168+0000 7fce55ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:03:51.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.168+0000 7fce55ffb640 1 -- 172.21.15.193:0/2219988032 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce601462b0 con 0x7fce60142d90 2024-03-25T18:03:51.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.168+0000 7fce55ffb640 10 monclient: _finish_auth 0 2024-03-25T18:03:51.170 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.168+0000 7fce55ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:03:51.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.168+0000 7fce55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:51.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.168+0000 7fce6792b640 5 monclient: authenticate success, global_id 4144 2024-03-25T18:03:51.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.168+0000 7fce6792b640 10 monclient: _renew_subs 2024-03-25T18:03:51.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.168+0000 7fce6792b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:03:51.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.168+0000 7fce6792b640 1 -- 172.21.15.193:0/2219988032 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce60137780 con 0x7fce60142d90 2024-03-25T18:03:51.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.168+0000 7fce6792b640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:03:51.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.168+0000 7fce6792b640 10 monclient: _renew_subs 2024-03-25T18:03:51.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.168+0000 7fce6792b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:03:51.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.168+0000 7fce6792b640 1 -- 172.21.15.193:0/2219988032 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce60145ee0 con 0x7fce60142d90 2024-03-25T18:03:51.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.168+0000 7fce6792b640 1 librados: init done 2024-03-25T18:03:51.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.169+0000 7fce6792b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:51.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.169+0000 7fce6792b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:51.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.169+0000 7fce6792b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:03:51.171 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.169+0000 7fce6792b640 1 -- 172.21.15.193:0/2219988032 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce60146440 con 0x7fce60142d90 2024-03-25T18:03:51.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.172+0000 7fce55ffb640 1 -- 172.21.15.193:0/2219988032 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce580016c0 con 0x7fce60142d90 2024-03-25T18:03:51.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.172+0000 7fce55ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:51.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.172+0000 7fce55ffb640 1 -- 172.21.15.193:0/2219988032 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fce58005190 con 0x7fce60142d90 2024-03-25T18:03:51.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.172+0000 7fce55ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:51.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.172+0000 7fce55ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:03:51.172 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.172+0000 7fce55ffb640 10 monclient: dump: 2024-03-25T18:03:51.173 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:51.173 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:51.173 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.173 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.173 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:51.173 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:51.173 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:51.173 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:51.173 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:51.173 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:51.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.172+0000 7fce55ffb640 1 -- 172.21.15.193:0/2219988032 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fce58005740 con 0x7fce60142d90 2024-03-25T18:03:51.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.172+0000 7fce577fe640 4 set_mon_vals no callback set 2024-03-25T18:03:51.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.172+0000 7fce55ffb640 1 -- 172.21.15.193:0/2219988032 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 59836222 0 0) 0x7fce58006e60 con 0x7fce60142d90 2024-03-25T18:03:51.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.172+0000 7fce55ffb640 10 client.4144.objecter ms_dispatch 0x7fce60000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T18:03:51.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.172+0000 7fce55ffb640 3 client.4144.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T18:03:51.173 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.172+0000 7fce55ffb640 3 client.4144.objecter handle_osd_map decoding full epoch 7 2024-03-25T18:03:51.174 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.172+0000 7fce55ffb640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-25T18:03:51.266 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.267+0000 7fce55ffb640 1 -- 172.21.15.193:0/2219988032 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fce580069f0 con 0x7fce60142d90 2024-03-25T18:03:51.266 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.267+0000 7fce55ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:51.266 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.267+0000 7fce55ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:03:51.351 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.351+0000 7fce6792b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "0ae5db93-018c-48ff-a423-7061d004c4c1", "id": 5}] 2024-03-25T18:03:51.351 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.351+0000 7fce6792b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "0ae5db93-018c-48ff-a423-7061d004c4c1", "id": 5}] 2024-03-25T18:03:51.351 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.351+0000 7fce6792b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:03:51.351 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.351+0000 7fce6792b640 1 -- 172.21.15.193:0/2219988032 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd new", "uuid": "0ae5db93-018c-48ff-a423-7061d004c4c1", "id": 5} v 0) v1 -- 0x7fce6004f730 con 0x7fce60142d90 2024-03-25T18:03:51.413 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.414+0000 7fce55ffb640 1 -- 172.21.15.193:0/2219988032 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0ae5db93-018c-48ff-a423-7061d004c4c1", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1727260452 0 1988412176) 0x7fce580059f0 con 0x7fce60142d90 2024-03-25T18:03:51.413 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.414+0000 7fce55ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "0ae5db93-018c-48ff-a423-7061d004c4c1", "id": 5}] 2024-03-25T18:03:51.414 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.414+0000 7fce55ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:03:51.414 INFO:teuthology.orchestra.run.smithi193.stdout:5 2024-03-25T18:03:51.415 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.415+0000 7fce6792b640 10 librados: watch_flush enter 2024-03-25T18:03:51.415 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.415+0000 7fce6792b640 10 librados: watch_flush exit 2024-03-25T18:03:51.415 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.415+0000 7fce6792b640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-25T18:03:51.415 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.415+0000 7fce6792b640 10 client.4144.objecter successfully canceled tick 2024-03-25T18:03:51.415 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.415+0000 7fce6792b640 10 monclient: shutdown 2024-03-25T18:03:51.415 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.415+0000 7fce6792b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:51.416 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.415+0000 7fce6792b640 1 -- 172.21.15.193:0/2219988032 >> v1:172.21.15.193:6789/0 conn(0x7fce60142d90 legacy=0x7fce601dae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:51.416 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.415+0000 7fce6792b640 1 -- 172.21.15.193:0/2219988032 shutdown_connections 2024-03-25T18:03:51.416 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.415+0000 7fce6792b640 1 -- 172.21.15.193:0/2219988032 >> 172.21.15.193:0/2219988032 conn(0x7fce600b5e40 msgr2=0x7fce600b6220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:51.416 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.415+0000 7fce6792b640 1 -- 172.21.15.193:0/2219988032 shutdown_connections 2024-03-25T18:03:51.416 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.415+0000 7fce6792b640 1 -- 172.21.15.193:0/2219988032 wait complete. 2024-03-25T18:03:51.416 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.415+0000 7fce6792b640 1 librados: shutdown 2024-03-25T18:03:51.425 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc 6 2024-03-25T18:03:51.456 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T18:03:51.456+0000 7f5e09feb040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:03:51.526 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.525+0000 7f81e948d640 10 monclient: get_monmap_and_config 2024-03-25T18:03:51.526 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.525+0000 7f81e948d640 10 monclient: build_initial_monmap 2024-03-25T18:03:51.526 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.525+0000 7f81e948d640 10 monclient: monmap: 2024-03-25T18:03:51.526 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T18:03:51.526 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:51.526 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:03:51.526194+0000 2024-03-25T18:03:51.526 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:03:51.526194+0000 2024-03-25T18:03:51.526 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:51.526 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:51.526 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:03:51.526 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:03:51.526 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:03:51.526 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:51.527 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.526+0000 7f81e948d640 1 Processor -- start 2024-03-25T18:03:51.527 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.526+0000 7f81e948d640 1 -- start start 2024-03-25T18:03:51.527 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.526+0000 7f81e948d640 10 monclient: init 2024-03-25T18:03:51.528 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.527+0000 7f81e948d640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:51.528 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.527+0000 7f81e948d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:03:51.528 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.527+0000 7f81e948d640 10 monclient(hunting): picked mon.noname-b con 0x7f81e40b4700 addr v1:172.21.15.66:6790/0 2024-03-25T18:03:51.528 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.527+0000 7f81e948d640 10 monclient(hunting): picked mon.noname-c con 0x7f81e4142a20 addr v1:172.21.15.193:6789/0 2024-03-25T18:03:51.528 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.527+0000 7f81e948d640 10 monclient(hunting): picked mon.noname-a con 0x7f81e40a96b0 addr v1:172.21.15.66:6789/0 2024-03-25T18:03:51.528 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.527+0000 7f81e948d640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e40a9d60 con 0x7f81e40a96b0 2024-03-25T18:03:51.528 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.527+0000 7f81e948d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e413ff10 con 0x7f81e4142a20 2024-03-25T18:03:51.528 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.527+0000 7f81e948d640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e40aa1f0 con 0x7f81e40b4700 2024-03-25T18:03:51.528 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.527+0000 7f81e948d640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:51.528 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.527+0000 7f81e948d640 10 monclient(hunting): authenticate will time out at 6891.860071s 2024-03-25T18:03:51.532 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.533+0000 7f81e37fe640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f81e40a96b0 0x7f81e413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.193:48022/0 (socket says 172.21.15.193:48022) 2024-03-25T18:03:51.532 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.533+0000 7f81e37fe640 1 -- 172.21.15.193:0/4146415543 learned_addr learned my addr 172.21.15.193:0/4146415543 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T18:03:51.546 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.544+0000 7f81e1ffb640 1 -- 172.21.15.193:0/4146415543 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590527543 0 0) 0x7f81e413ff10 con 0x7f81e4142a20 2024-03-25T18:03:51.546 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.544+0000 7f81e1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:03:51.546 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.544+0000 7f81e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:51.546 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.544+0000 7f81e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:51.546 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.544+0000 7f81e1ffb640 10 monclient(hunting): my global_id is 4147 2024-03-25T18:03:51.546 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.544+0000 7f81e1ffb640 1 -- 172.21.15.193:0/4146415543 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81cc003520 con 0x7f81e4142a20 2024-03-25T18:03:51.546 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.544+0000 7f81e1ffb640 1 -- 172.21.15.193:0/4146415543 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4127130306 0 0) 0x7f81e40aa1f0 con 0x7f81e40b4700 2024-03-25T18:03:51.546 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.544+0000 7f81e1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:03:51.546 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.544+0000 7f81e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:51.546 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.544+0000 7f81e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:51.547 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.544+0000 7f81e1ffb640 10 monclient(hunting): my global_id is 4148 2024-03-25T18:03:51.547 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.544+0000 7f81e1ffb640 1 -- 172.21.15.193:0/4146415543 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81e413ff10 con 0x7f81e40b4700 2024-03-25T18:03:51.547 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.545+0000 7f81e1ffb640 1 -- 172.21.15.193:0/4146415543 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1013207320 0 0) 0x7f81e40a9d60 con 0x7f81e40a96b0 2024-03-25T18:03:51.547 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.545+0000 7f81e1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:03:51.547 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.545+0000 7f81e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:51.547 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.545+0000 7f81e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:51.547 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.545+0000 7f81e1ffb640 10 monclient(hunting): my global_id is 4152 2024-03-25T18:03:51.547 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.545+0000 7f81e1ffb640 1 -- 172.21.15.193:0/4146415543 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81e40aa1f0 con 0x7f81e40a96b0 2024-03-25T18:03:51.548 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.548+0000 7f81e1ffb640 1 -- 172.21.15.193:0/4146415543 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2105683132 0 0) 0x7f81cc003520 con 0x7f81e4142a20 2024-03-25T18:03:51.548 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.548+0000 7f81e1ffb640 1 -- 172.21.15.193:0/4146415543 >> v1:172.21.15.66:6790/0 conn(0x7f81e40b4700 legacy=0x7f81e4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:51.548 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.548+0000 7f81e1ffb640 1 -- 172.21.15.193:0/4146415543 >> v1:172.21.15.66:6789/0 conn(0x7f81e40a96b0 legacy=0x7f81e413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:51.548 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.548+0000 7f81e1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:03:51.548 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.548+0000 7f81e1ffb640 1 monclient: found mon.noname-c 2024-03-25T18:03:51.548 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.548+0000 7f81e1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:51.548 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.548+0000 7f81e1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:03:51.548 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.548+0000 7f81e1ffb640 1 -- 172.21.15.193:0/4146415543 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e40b4ae0 con 0x7f81e4142a20 2024-03-25T18:03:51.548 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.548+0000 7f81e1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:03:51.548 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.548+0000 7f81e1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:03:51.549 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.548+0000 7f81e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:51.549 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.548+0000 7f81e1ffb640 1 -- 172.21.15.193:0/4146415543 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f81d8001690 con 0x7f81e4142a20 2024-03-25T18:03:51.549 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.548+0000 7f81e948d640 5 monclient: authenticate success, global_id 4147 2024-03-25T18:03:51.549 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.548+0000 7f81e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:51.549 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.548+0000 7f81e1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:03:51.549 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.548+0000 7f81e1ffb640 10 monclient: dump: 2024-03-25T18:03:51.549 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:51.549 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:51.549 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.549 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.549 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:51.549 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:51.549 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:51.549 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:51.549 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:51.549 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:51.550 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.548+0000 7f81e948d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:51.551 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.550+0000 7f81e1ffb640 1 -- 172.21.15.193:0/4146415543 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81d80017f0 con 0x7f81e4142a20 2024-03-25T18:03:51.551 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.550+0000 7f81e1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:51.551 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.550+0000 7f81e1ffb640 1 -- 172.21.15.193:0/4146415543 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f81d8001c00 con 0x7f81e4142a20 2024-03-25T18:03:51.551 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.550+0000 7f81e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:51.551 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.550+0000 7f81e1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:03:51.551 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.550+0000 7f81e1ffb640 10 monclient: dump: 2024-03-25T18:03:51.552 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:51.552 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:51.552 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.552 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.552 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:51.552 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:51.552 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:51.552 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:51.552 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:51.552 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:51.552 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.550+0000 7f81e948d640 10 monclient: get_monmap_and_config success 2024-03-25T18:03:51.552 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.550+0000 7f81e948d640 4 set_mon_vals no callback set 2024-03-25T18:03:51.552 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.550+0000 7f81e948d640 10 monclient: shutdown 2024-03-25T18:03:51.552 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.550+0000 7f81e948d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:51.552 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.550+0000 7f81e948d640 1 -- 172.21.15.193:0/4146415543 >> v1:172.21.15.193:6789/0 conn(0x7f81e4142a20 legacy=0x7f81e413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:51.553 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.550+0000 7f81e948d640 1 -- 172.21.15.193:0/4146415543 shutdown_connections 2024-03-25T18:03:51.553 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.550+0000 7f81e948d640 1 -- 172.21.15.193:0/4146415543 >> 172.21.15.193:0/4146415543 conn(0x7f81e40668e0 msgr2=0x7f81e4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:51.553 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.550+0000 7f81e948d640 1 -- 172.21.15.193:0/4146415543 shutdown_connections 2024-03-25T18:03:51.553 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.550+0000 7f81e948d640 1 -- 172.21.15.193:0/4146415543 wait complete. 2024-03-25T18:03:51.553 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.551+0000 7f81e948d640 10 monclient: build_initial_monmap 2024-03-25T18:03:51.553 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.551+0000 7f81e948d640 10 monclient: monmap: 2024-03-25T18:03:51.553 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T18:03:51.553 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:03:51.553 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 0.000000 2024-03-25T18:03:51.553 INFO:teuthology.orchestra.run.smithi193.stderr:created 0.000000 2024-03-25T18:03:51.553 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:51.554 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 1 2024-03-25T18:03:51.554 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:03:51.554 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:03:51.554 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:03:51.554 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:51.554 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.551+0000 7f81e948d640 1 librados: starting msgr at 2024-03-25T18:03:51.554 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.551+0000 7f81e948d640 1 librados: starting objecter 2024-03-25T18:03:51.554 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.552+0000 7f81e948d640 1 Processor -- start 2024-03-25T18:03:51.554 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.552+0000 7f81e948d640 1 -- start start 2024-03-25T18:03:51.554 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.552+0000 7f81e948d640 1 librados: setting wanted keys 2024-03-25T18:03:51.554 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.552+0000 7f81e948d640 1 librados: calling monclient init 2024-03-25T18:03:51.554 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.552+0000 7f81e948d640 10 monclient: init 2024-03-25T18:03:51.556 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.552+0000 7f81e948d640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:51.556 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.552+0000 7f81e948d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:03:51.557 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.552+0000 7f81e948d640 10 monclient(hunting): picked mon.noname-a con 0x7f81e40a96b0 addr v1:172.21.15.66:6789/0 2024-03-25T18:03:51.557 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.552+0000 7f81e948d640 10 monclient(hunting): picked mon.noname-c con 0x7f81e40b4700 addr v1:172.21.15.66:6790/0 2024-03-25T18:03:51.557 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.552+0000 7f81e948d640 10 monclient(hunting): picked mon.noname-b con 0x7f81e4142a20 addr v1:172.21.15.193:6789/0 2024-03-25T18:03:51.557 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.552+0000 7f81e948d640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e40ac1b0 con 0x7f81e40a96b0 2024-03-25T18:03:51.557 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.552+0000 7f81e948d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e413a640 con 0x7f81e4142a20 2024-03-25T18:03:51.557 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.552+0000 7f81e948d640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e413c430 con 0x7f81e40b4700 2024-03-25T18:03:51.557 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.552+0000 7f81e948d640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:51.557 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.552+0000 7f81e948d640 10 monclient(hunting): authenticate will time out at 6891.885467s 2024-03-25T18:03:51.557 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.553+0000 7f81e27fc640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f81e40b4700 0x7f81e4138870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.193:44954/0 (socket says 172.21.15.193:44954) 2024-03-25T18:03:51.557 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.553+0000 7f81e27fc640 1 -- 172.21.15.193:0/2123708474 learned_addr learned my addr 172.21.15.193:0/2123708474 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T18:03:51.557 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.553+0000 7f81cb7fe640 10 client.?.objecter ms_handle_connect 0x7f81e40b4700 2024-03-25T18:03:51.557 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.553+0000 7f81cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:51.557 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.553+0000 7f81cb7fe640 10 client.?.objecter ms_handle_connect 0x7f81e40a96b0 2024-03-25T18:03:51.557 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.553+0000 7f81cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:51.557 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.553+0000 7f81cb7fe640 10 client.?.objecter ms_handle_connect 0x7f81e4142a20 2024-03-25T18:03:51.558 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.553+0000 7f81cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:51.558 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.555+0000 7f81cb7fe640 1 -- 172.21.15.193:0/2123708474 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2471794216 0 0) 0x7f81e413c430 con 0x7f81e40b4700 2024-03-25T18:03:51.558 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.555+0000 7f81cb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:03:51.558 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.555+0000 7f81cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:51.558 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.555+0000 7f81cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:51.558 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.555+0000 7f81cb7fe640 10 monclient(hunting): my global_id is 4151 2024-03-25T18:03:51.558 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.555+0000 7f81cb7fe640 1 -- 172.21.15.193:0/2123708474 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81d0004eb0 con 0x7f81e40b4700 2024-03-25T18:03:51.558 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.555+0000 7f81cb7fe640 1 -- 172.21.15.193:0/2123708474 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2685212954 0 0) 0x7f81e40ac1b0 con 0x7f81e40a96b0 2024-03-25T18:03:51.558 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.556+0000 7f81cb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:03:51.558 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.556+0000 7f81cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:51.558 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.556+0000 7f81cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:51.558 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.556+0000 7f81cb7fe640 10 monclient(hunting): my global_id is 4155 2024-03-25T18:03:51.558 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.556+0000 7f81cb7fe640 1 -- 172.21.15.193:0/2123708474 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81e413c430 con 0x7f81e40a96b0 2024-03-25T18:03:51.558 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.556+0000 7f81cb7fe640 1 -- 172.21.15.193:0/2123708474 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662009496 0 0) 0x7f81e413a640 con 0x7f81e4142a20 2024-03-25T18:03:51.559 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.556+0000 7f81cb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:03:51.559 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.556+0000 7f81cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:51.559 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.556+0000 7f81cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:51.559 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.556+0000 7f81cb7fe640 10 monclient(hunting): my global_id is 4150 2024-03-25T18:03:51.559 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.556+0000 7f81cb7fe640 1 -- 172.21.15.193:0/2123708474 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81e40ac1b0 con 0x7f81e4142a20 2024-03-25T18:03:51.561 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.561+0000 7f81cb7fe640 1 -- 172.21.15.193:0/2123708474 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 647313565 0 0) 0x7f81d0004eb0 con 0x7f81e40b4700 2024-03-25T18:03:51.561 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.561+0000 7f81cb7fe640 1 -- 172.21.15.193:0/2123708474 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81e413a640 con 0x7f81e40b4700 2024-03-25T18:03:51.562 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.561+0000 7f81cb7fe640 1 -- 172.21.15.193:0/2123708474 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f81d8001bc0 con 0x7f81e40b4700 2024-03-25T18:03:51.562 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.561+0000 7f81cb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:51.562 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.561+0000 7f81cb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:03:51.562 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.561+0000 7f81cb7fe640 10 monclient(hunting): dump: 2024-03-25T18:03:51.562 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:51.562 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:51.562 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.562 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.562 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:51.562 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:51.562 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:51.562 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:51.562 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:51.562 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:51.563 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.561+0000 7f81cb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:03:51.563 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.561+0000 7f81cb7fe640 1 -- 172.21.15.193:0/2123708474 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3500876438 0 0) 0x7f81e413c430 con 0x7f81e40a96b0 2024-03-25T18:03:51.563 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.561+0000 7f81cb7fe640 1 -- 172.21.15.193:0/2123708474 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81d0004eb0 con 0x7f81e40a96b0 2024-03-25T18:03:51.563 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.561+0000 7f81cb7fe640 1 -- 172.21.15.193:0/2123708474 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f81d4001970 con 0x7f81e40a96b0 2024-03-25T18:03:51.563 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.561+0000 7f81cb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:51.563 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.561+0000 7f81cb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:03:51.563 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.561+0000 7f81cb7fe640 10 monclient(hunting): dump: 2024-03-25T18:03:51.563 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:51.563 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:51.563 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.563 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.563 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:51.563 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:51.564 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:51.564 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:51.564 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:51.564 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:51.564 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.561+0000 7f81cb7fe640 1 -- 172.21.15.193:0/2123708474 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2762374061 0 0) 0x7f81e40ac1b0 con 0x7f81e4142a20 2024-03-25T18:03:51.564 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.561+0000 7f81cb7fe640 1 -- 172.21.15.193:0/2123708474 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81e413c430 con 0x7f81e4142a20 2024-03-25T18:03:51.564 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.561+0000 7f81cb7fe640 1 -- 172.21.15.193:0/2123708474 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f81dc001bd0 con 0x7f81e4142a20 2024-03-25T18:03:51.564 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.561+0000 7f81cb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:51.564 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.561+0000 7f81cb7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:03:51.564 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.561+0000 7f81cb7fe640 10 monclient(hunting): dump: 2024-03-25T18:03:51.564 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:51.564 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:51.564 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.564 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.565 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:51.565 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:51.565 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:51.565 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:51.565 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:51.565 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:51.566 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.564+0000 7f81cb7fe640 1 -- 172.21.15.193:0/2123708474 <== mon.2 v1:172.21.15.66:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 137517646 0 0) 0x7f81e413a640 con 0x7f81e40b4700 2024-03-25T18:03:51.566 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.564+0000 7f81cb7fe640 1 -- 172.21.15.193:0/2123708474 >> v1:172.21.15.193:6789/0 conn(0x7f81e4142a20 legacy=0x7f81e41e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:51.567 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.564+0000 7f81cb7fe640 1 -- 172.21.15.193:0/2123708474 >> v1:172.21.15.66:6789/0 conn(0x7f81e40a96b0 legacy=0x7f81e413bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:51.567 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.564+0000 7f81cb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:03:51.567 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.564+0000 7f81cb7fe640 1 monclient: found mon.c 2024-03-25T18:03:51.567 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.564+0000 7f81cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:51.567 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.564+0000 7f81cb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:03:51.567 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.564+0000 7f81cb7fe640 1 -- 172.21.15.193:0/2123708474 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e413c5c0 con 0x7f81e40b4700 2024-03-25T18:03:51.567 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.564+0000 7f81cb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:03:51.567 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.564+0000 7f81cb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:03:51.567 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.564+0000 7f81cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:51.567 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.564+0000 7f81e948d640 5 monclient: authenticate success, global_id 4151 2024-03-25T18:03:51.567 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.564+0000 7f81e948d640 10 monclient: _renew_subs 2024-03-25T18:03:51.567 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.564+0000 7f81e948d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:03:51.567 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.564+0000 7f81e948d640 1 -- 172.21.15.193:0/2123708474 --> v1:172.21.15.66:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81e413b080 con 0x7f81e40b4700 2024-03-25T18:03:51.567 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.564+0000 7f81e948d640 10 client.4151.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:03:51.567 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.564+0000 7f81e948d640 10 monclient: _renew_subs 2024-03-25T18:03:51.568 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.564+0000 7f81e948d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:03:51.568 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.564+0000 7f81e948d640 1 -- 172.21.15.193:0/2123708474 --> v1:172.21.15.66:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81e40ac720 con 0x7f81e40b4700 2024-03-25T18:03:51.568 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.564+0000 7f81e948d640 1 librados: init done 2024-03-25T18:03:51.568 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.565+0000 7f81e948d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:51.568 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.565+0000 7f81e948d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:51.568 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.565+0000 7f81e948d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:03:51.568 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.565+0000 7f81e948d640 1 -- 172.21.15.193:0/2123708474 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81e413b380 con 0x7f81e40b4700 2024-03-25T18:03:51.661 INFO:tasks.ceph.mgr.y.smithi066.stderr:2024-03-25T18:03:51.661+0000 7fc1b688f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:03:51.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.666+0000 7f81cb7fe640 1 -- 172.21.15.193:0/2123708474 <== mon.2 v1:172.21.15.66:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81d80011f0 con 0x7f81e40b4700 2024-03-25T18:03:51.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.667+0000 7f81cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:51.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.667+0000 7f81cb7fe640 1 -- 172.21.15.193:0/2123708474 <== mon.2 v1:172.21.15.66:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f81d8003b00 con 0x7f81e40b4700 2024-03-25T18:03:51.666 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.667+0000 7f81cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:51.667 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.667+0000 7f81cb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:03:51.667 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.667+0000 7f81cb7fe640 10 monclient: dump: 2024-03-25T18:03:51.667 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:51.667 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:51.667 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.667 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.667 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:51.667 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:51.667 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:51.667 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:51.667 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:51.667 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:51.667 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.667+0000 7f81cb7fe640 1 -- 172.21.15.193:0/2123708474 <== mon.2 v1:172.21.15.66:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f81d80040b0 con 0x7f81e40b4700 2024-03-25T18:03:51.667 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.667+0000 7f81e17fa640 4 set_mon_vals no callback set 2024-03-25T18:03:51.668 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.667+0000 7f81cb7fe640 1 -- 172.21.15.193:0/2123708474 <== mon.2 v1:172.21.15.66:6790/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 1902304476 0 0) 0x7f81d8006900 con 0x7f81e40b4700 2024-03-25T18:03:51.668 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.667+0000 7f81cb7fe640 10 client.4151.objecter ms_dispatch 0x7f81e4000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T18:03:51.668 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.667+0000 7f81cb7fe640 3 client.4151.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T18:03:51.668 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.667+0000 7f81cb7fe640 3 client.4151.objecter handle_osd_map decoding full epoch 8 2024-03-25T18:03:51.668 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.668+0000 7f81cb7fe640 20 client.4151.objecter dump_active .. 0 homeless 2024-03-25T18:03:51.668 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.668+0000 7f81cb7fe640 1 -- 172.21.15.193:0/2123708474 <== mon.2 v1:172.21.15.66:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f81d8006310 con 0x7f81e40b4700 2024-03-25T18:03:51.668 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.668+0000 7f81cb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:51.669 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.668+0000 7f81cb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:03:51.739 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.739+0000 7f81e948d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc", "id": 6}] 2024-03-25T18:03:51.739 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.739+0000 7f81e948d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc", "id": 6}] 2024-03-25T18:03:51.739 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.739+0000 7f81e948d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:03:51.739 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.739+0000 7f81e948d640 1 -- 172.21.15.193:0/2123708474 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "osd new", "uuid": "c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc", "id": 6} v 0) v1 -- 0x7f81e413a7d0 con 0x7f81e40b4700 2024-03-25T18:03:51.744 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T18:03:51.744+0000 7f5e09feb040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:03:51.796 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T18:03:51.796+0000 7f5e09feb040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:03:51.815 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.815+0000 7f81cb7fe640 1 -- 172.21.15.193:0/2123708474 <== mon.2 v1:172.21.15.66:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 1099858463 0 1113811849) 0x7f81d8006660 con 0x7f81e40b4700 2024-03-25T18:03:51.815 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.815+0000 7f81cb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc", "id": 6}] 2024-03-25T18:03:51.815 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.815+0000 7f81cb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:03:51.815 INFO:teuthology.orchestra.run.smithi193.stdout:6 2024-03-25T18:03:51.817 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.816+0000 7f81e948d640 10 librados: watch_flush enter 2024-03-25T18:03:51.817 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.816+0000 7f81e948d640 10 librados: watch_flush exit 2024-03-25T18:03:51.817 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.816+0000 7f81e948d640 20 client.4151.objecter shutdown clearing up homeless session... 2024-03-25T18:03:51.817 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.816+0000 7f81e948d640 10 client.4151.objecter successfully canceled tick 2024-03-25T18:03:51.817 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.816+0000 7f81e948d640 10 monclient: shutdown 2024-03-25T18:03:51.817 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.816+0000 7f81e948d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:51.817 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.816+0000 7f81e948d640 1 -- 172.21.15.193:0/2123708474 >> v1:172.21.15.66:6790/0 conn(0x7f81e40b4700 legacy=0x7f81e4138870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:51.817 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.817+0000 7f81e948d640 1 -- 172.21.15.193:0/2123708474 shutdown_connections 2024-03-25T18:03:51.817 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.817+0000 7f81e948d640 1 -- 172.21.15.193:0/2123708474 >> 172.21.15.193:0/2123708474 conn(0x7f81e40668e0 msgr2=0x7f81e413e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:51.817 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.817+0000 7f81e948d640 1 -- 172.21.15.193:0/2123708474 shutdown_connections 2024-03-25T18:03:51.817 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.817+0000 7f81e948d640 1 -- 172.21.15.193:0/2123708474 wait complete. 2024-03-25T18:03:51.817 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.817+0000 7f81e948d640 1 librados: shutdown 2024-03-25T18:03:51.827 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new 59a326e8-ae37-42fc-a22c-24b55a726954 7 2024-03-25T18:03:51.917 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T18:03:51.917+0000 7f5e09feb040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:03:51.955 INFO:tasks.ceph.mgr.y.smithi066.stderr:2024-03-25T18:03:51.956+0000 7fc1b688f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:03:51.967 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.967+0000 7f41f8674640 10 monclient: get_monmap_and_config 2024-03-25T18:03:51.967 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.967+0000 7f41f8674640 10 monclient: build_initial_monmap 2024-03-25T18:03:51.967 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.967+0000 7f41f8674640 10 monclient: monmap: 2024-03-25T18:03:51.967 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T18:03:51.967 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:51.967 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:03:51.967971+0000 2024-03-25T18:03:51.968 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:03:51.967971+0000 2024-03-25T18:03:51.968 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:51.968 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:51.968 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:03:51.968 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:03:51.968 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:03:51.968 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:51.970 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.968+0000 7f41f8674640 1 Processor -- start 2024-03-25T18:03:51.970 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.968+0000 7f41f8674640 1 -- start start 2024-03-25T18:03:51.970 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.968+0000 7f41f8674640 10 monclient: init 2024-03-25T18:03:51.970 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.968+0000 7f41f8674640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:51.970 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.968+0000 7f41f8674640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:03:51.970 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.968+0000 7f41f8674640 10 monclient(hunting): picked mon.noname-a con 0x7f41f00a9710 addr v1:172.21.15.66:6789/0 2024-03-25T18:03:51.970 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.968+0000 7f41f8674640 10 monclient(hunting): picked mon.noname-b con 0x7f41f00aafa0 addr v1:172.21.15.66:6790/0 2024-03-25T18:03:51.970 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.969+0000 7f41f8674640 10 monclient(hunting): picked mon.noname-c con 0x7f41f013c570 addr v1:172.21.15.193:6789/0 2024-03-25T18:03:51.970 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.969+0000 7f41f8674640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f00ab420 con 0x7f41f00a9710 2024-03-25T18:03:51.970 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.969+0000 7f41f8674640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f00b0190 con 0x7f41f013c570 2024-03-25T18:03:51.970 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.969+0000 7f41f8674640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f00abdb0 con 0x7f41f00aafa0 2024-03-25T18:03:51.970 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.969+0000 7f41f8674640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:51.971 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.969+0000 7f41f8674640 10 monclient(hunting): authenticate will time out at 6892.301861s 2024-03-25T18:03:51.971 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.969+0000 7f41f6c11640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f41f013c570 0x7f41f014a9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:47278/0 (socket says 172.21.15.193:47278) 2024-03-25T18:03:51.971 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.969+0000 7f41f6c11640 1 -- 172.21.15.193:0/3453250286 learned_addr learned my addr 172.21.15.193:0/3453250286 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T18:03:51.972 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.971+0000 7f41f540e640 1 -- 172.21.15.193:0/3453250286 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3727128644 0 0) 0x7f41f00b0190 con 0x7f41f013c570 2024-03-25T18:03:51.972 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.971+0000 7f41f540e640 10 monclient(hunting): _negotiate 2024-03-25T18:03:51.972 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.971+0000 7f41f540e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:51.972 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.971+0000 7f41f540e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:51.972 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.971+0000 7f41f540e640 10 monclient(hunting): my global_id is 4153 2024-03-25T18:03:51.972 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.971+0000 7f41f540e640 1 -- 172.21.15.193:0/3453250286 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41dc005730 con 0x7f41f013c570 2024-03-25T18:03:51.972 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.972+0000 7f41f540e640 1 -- 172.21.15.193:0/3453250286 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2300815529 0 0) 0x7f41f00abdb0 con 0x7f41f00aafa0 2024-03-25T18:03:51.972 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.972+0000 7f41f540e640 10 monclient(hunting): _negotiate 2024-03-25T18:03:51.972 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.972+0000 7f41f540e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:51.972 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.972+0000 7f41f540e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:51.973 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.972+0000 7f41f540e640 10 monclient(hunting): my global_id is 4154 2024-03-25T18:03:51.973 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.972+0000 7f41f540e640 1 -- 172.21.15.193:0/3453250286 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41f00b0190 con 0x7f41f00aafa0 2024-03-25T18:03:51.975 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.975+0000 7f41f540e640 1 -- 172.21.15.193:0/3453250286 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4000654675 0 0) 0x7f41dc005730 con 0x7f41f013c570 2024-03-25T18:03:51.975 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.975+0000 7f41f540e640 1 -- 172.21.15.193:0/3453250286 >> v1:172.21.15.66:6790/0 conn(0x7f41f00aafa0 legacy=0x7f41f00724a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:51.975 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.975+0000 7f41f540e640 1 -- 172.21.15.193:0/3453250286 >> v1:172.21.15.66:6789/0 conn(0x7f41f00a9710 legacy=0x7f41f00ad3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:03:51.975 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.975+0000 7f41f540e640 10 monclient: _finish_hunting 0 2024-03-25T18:03:51.975 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.975+0000 7f41f540e640 1 monclient: found mon.noname-c 2024-03-25T18:03:51.975 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.975+0000 7f41f540e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:51.975 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.975+0000 7f41f540e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:03:51.975 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.975+0000 7f41f540e640 1 -- 172.21.15.193:0/3453250286 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f00abf40 con 0x7f41f013c570 2024-03-25T18:03:51.975 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.975+0000 7f41f540e640 10 monclient: _finish_auth 0 2024-03-25T18:03:51.975 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.975+0000 7f41f540e640 10 monclient: _check_auth_tickets 2024-03-25T18:03:51.975 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.975+0000 7f41f540e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:51.976 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.975+0000 7f41f540e640 1 -- 172.21.15.193:0/3453250286 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f41e8001a10 con 0x7f41f013c570 2024-03-25T18:03:51.976 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.975+0000 7f41f540e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:51.976 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.975+0000 7f41f540e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:03:51.976 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.975+0000 7f41f540e640 10 monclient: dump: 2024-03-25T18:03:51.976 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:51.976 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:51.976 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.976 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.976 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:51.976 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:51.976 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:51.976 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:51.977 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:51.977 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:51.977 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.975+0000 7f41f8674640 5 monclient: authenticate success, global_id 4153 2024-03-25T18:03:51.977 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.975+0000 7f41f8674640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:51.979 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.976+0000 7f41f540e640 1 -- 172.21.15.193:0/3453250286 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41e8001b70 con 0x7f41f013c570 2024-03-25T18:03:51.980 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.976+0000 7f41f540e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:51.980 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.976+0000 7f41f540e640 1 -- 172.21.15.193:0/3453250286 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f41e8001f80 con 0x7f41f013c570 2024-03-25T18:03:51.980 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.976+0000 7f41f540e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:51.980 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.976+0000 7f41f540e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:03:51.980 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.976+0000 7f41f540e640 10 monclient: dump: 2024-03-25T18:03:51.980 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:51.980 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:51.980 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.980 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.980 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:51.980 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:51.980 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:51.980 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:51.980 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:51.981 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:51.981 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.976+0000 7f41f8674640 10 monclient: get_monmap_and_config success 2024-03-25T18:03:51.981 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.976+0000 7f41f8674640 4 set_mon_vals no callback set 2024-03-25T18:03:51.981 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.977+0000 7f41f8674640 10 monclient: shutdown 2024-03-25T18:03:51.981 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.977+0000 7f41f8674640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:51.981 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.977+0000 7f41f8674640 1 -- 172.21.15.193:0/3453250286 >> v1:172.21.15.193:6789/0 conn(0x7f41f013c570 legacy=0x7f41f014a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:51.981 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.977+0000 7f41f8674640 1 -- 172.21.15.193:0/3453250286 shutdown_connections 2024-03-25T18:03:51.981 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.977+0000 7f41f8674640 1 -- 172.21.15.193:0/3453250286 >> 172.21.15.193:0/3453250286 conn(0x7f41f006b560 msgr2=0x7f41f00b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:51.981 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.977+0000 7f41f8674640 1 -- 172.21.15.193:0/3453250286 shutdown_connections 2024-03-25T18:03:51.981 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.977+0000 7f41f8674640 1 -- 172.21.15.193:0/3453250286 wait complete. 2024-03-25T18:03:51.981 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.978+0000 7f41f8674640 10 monclient: build_initial_monmap 2024-03-25T18:03:51.981 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.978+0000 7f41f8674640 10 monclient: monmap: 2024-03-25T18:03:51.981 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 0 2024-03-25T18:03:51.981 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:03:51.982 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 0.000000 2024-03-25T18:03:51.982 INFO:teuthology.orchestra.run.smithi193.stderr:created 0.000000 2024-03-25T18:03:51.982 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:51.982 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 1 2024-03-25T18:03:51.982 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:03:51.982 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:03:51.982 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:03:51.982 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:51.982 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.978+0000 7f41f8674640 1 librados: starting msgr at 2024-03-25T18:03:51.982 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.978+0000 7f41f8674640 1 librados: starting objecter 2024-03-25T18:03:51.982 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.978+0000 7f41f8674640 1 Processor -- start 2024-03-25T18:03:51.982 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.978+0000 7f41f8674640 1 -- start start 2024-03-25T18:03:51.982 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.978+0000 7f41f8674640 1 librados: setting wanted keys 2024-03-25T18:03:51.982 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.978+0000 7f41f8674640 1 librados: calling monclient init 2024-03-25T18:03:51.983 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.978+0000 7f41f8674640 10 monclient: init 2024-03-25T18:03:51.983 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.979+0000 7f41f8674640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:51.983 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.979+0000 7f41f8674640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:03:51.983 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.979+0000 7f41f8674640 10 monclient(hunting): picked mon.noname-a con 0x7f41f00a9710 addr v1:172.21.15.66:6789/0 2024-03-25T18:03:51.983 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.979+0000 7f41f8674640 10 monclient(hunting): picked mon.noname-b con 0x7f41f00aafa0 addr v1:172.21.15.193:6789/0 2024-03-25T18:03:51.983 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.979+0000 7f41f8674640 10 monclient(hunting): picked mon.noname-c con 0x7f41f013c570 addr v1:172.21.15.66:6790/0 2024-03-25T18:03:51.983 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.979+0000 7f41f8674640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f0141f20 con 0x7f41f00a9710 2024-03-25T18:03:51.983 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.979+0000 7f41f8674640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f0142b10 con 0x7f41f00aafa0 2024-03-25T18:03:51.983 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.979+0000 7f41f8674640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f0142150 con 0x7f41f013c570 2024-03-25T18:03:51.983 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.979+0000 7f41f8674640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:51.983 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.979+0000 7f41f8674640 10 monclient(hunting): authenticate will time out at 6892.311903s 2024-03-25T18:03:51.983 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.979+0000 7f41f5c0f640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f41f00aafa0 0x7f41f01d7280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:47288/0 (socket says 172.21.15.193:47288) 2024-03-25T18:03:51.983 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.979+0000 7f41f5c0f640 1 -- 172.21.15.193:0/3656875532 learned_addr learned my addr 172.21.15.193:0/3656875532 (peer_addr_for_me v1:172.21.15.193:0/0) 2024-03-25T18:03:51.983 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.979+0000 7f41e6ffd640 10 client.?.objecter ms_handle_connect 0x7f41f00aafa0 2024-03-25T18:03:51.983 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.979+0000 7f41e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:51.983 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.979+0000 7f41e6ffd640 10 client.?.objecter ms_handle_connect 0x7f41f013c570 2024-03-25T18:03:51.984 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.979+0000 7f41e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:51.984 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.981+0000 7f41e6ffd640 1 -- 172.21.15.193:0/3656875532 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3152994492 0 0) 0x7f41f0142b10 con 0x7f41f00aafa0 2024-03-25T18:03:51.984 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.981+0000 7f41e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:03:51.984 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.981+0000 7f41e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:51.984 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.981+0000 7f41e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:51.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.981+0000 7f41e6ffd640 10 monclient(hunting): my global_id is 4156 2024-03-25T18:03:51.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.981+0000 7f41e6ffd640 1 -- 172.21.15.193:0/3656875532 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41d4004eb0 con 0x7f41f00aafa0 2024-03-25T18:03:51.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.982+0000 7f41e6ffd640 1 -- 172.21.15.193:0/3656875532 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2944423027 0 0) 0x7f41f0142150 con 0x7f41f013c570 2024-03-25T18:03:51.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.982+0000 7f41e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:03:51.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.982+0000 7f41e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:51.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.982+0000 7f41e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:51.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.982+0000 7f41e6ffd640 10 monclient(hunting): my global_id is 4157 2024-03-25T18:03:51.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.982+0000 7f41e6ffd640 1 -- 172.21.15.193:0/3656875532 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41f0142b10 con 0x7f41f013c570 2024-03-25T18:03:51.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.986+0000 7f41e6ffd640 1 -- 172.21.15.193:0/3656875532 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4259633342 0 0) 0x7f41d4004eb0 con 0x7f41f00aafa0 2024-03-25T18:03:51.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.986+0000 7f41e6ffd640 1 -- 172.21.15.193:0/3656875532 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41f0142150 con 0x7f41f00aafa0 2024-03-25T18:03:51.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.986+0000 7f41e6ffd640 1 -- 172.21.15.193:0/3656875532 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f41e0001a10 con 0x7f41f00aafa0 2024-03-25T18:03:51.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.986+0000 7f41e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:51.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.986+0000 7f41e6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:03:51.986 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.986+0000 7f41e6ffd640 10 monclient(hunting): dump: 2024-03-25T18:03:51.986 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:51.986 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:51.986 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.986 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.986 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:51.987 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:51.987 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:51.987 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:51.987 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:51.987 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:51.987 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.986+0000 7f41e6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:03:51.987 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.987+0000 7f41e6ffd640 1 -- 172.21.15.193:0/3656875532 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1205049912 0 0) 0x7f41f0142b10 con 0x7f41f013c570 2024-03-25T18:03:51.987 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.987+0000 7f41e6ffd640 1 -- 172.21.15.193:0/3656875532 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41d4004eb0 con 0x7f41f013c570 2024-03-25T18:03:51.987 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.987+0000 7f41e6ffd640 1 -- 172.21.15.193:0/3656875532 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f41e8001e80 con 0x7f41f013c570 2024-03-25T18:03:51.988 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.987+0000 7f41e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:51.988 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.987+0000 7f41e6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:03:51.988 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.987+0000 7f41e6ffd640 10 monclient(hunting): dump: 2024-03-25T18:03:51.988 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:51.988 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:51.988 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.988 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:51.988 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:51.988 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:51.988 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:51.988 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:51.988 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:51.988 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:51.990 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.989+0000 7f41e6ffd640 1 -- 172.21.15.193:0/3656875532 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1928078153 0 0) 0x7f41f0142150 con 0x7f41f00aafa0 2024-03-25T18:03:51.990 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.989+0000 7f41e6ffd640 1 -- 172.21.15.193:0/3656875532 >> v1:172.21.15.66:6790/0 conn(0x7f41f013c570 legacy=0x7f41f01d46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:51.990 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.989+0000 7f41e6ffd640 1 -- 172.21.15.193:0/3656875532 >> v1:172.21.15.66:6789/0 conn(0x7f41f00a9710 legacy=0x7f41f0140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:51.990 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.989+0000 7f41e6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:03:51.990 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.989+0000 7f41e6ffd640 1 monclient: found mon.b 2024-03-25T18:03:51.990 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.989+0000 7f41e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:51.990 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.989+0000 7f41e6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:03:51.990 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.989+0000 7f41e6ffd640 1 -- 172.21.15.193:0/3656875532 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f01d4db0 con 0x7f41f00aafa0 2024-03-25T18:03:51.990 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.989+0000 7f41e6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:03:51.990 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.989+0000 7f41e6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:03:51.990 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.989+0000 7f41e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:51.990 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.989+0000 7f41f8674640 5 monclient: authenticate success, global_id 4156 2024-03-25T18:03:51.990 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.989+0000 7f41f8674640 10 monclient: _renew_subs 2024-03-25T18:03:51.990 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.989+0000 7f41f8674640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:03:51.991 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.989+0000 7f41f8674640 1 -- 172.21.15.193:0/3656875532 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41f01396b0 con 0x7f41f00aafa0 2024-03-25T18:03:51.991 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.989+0000 7f41f8674640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:03:51.991 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.989+0000 7f41f8674640 10 monclient: _renew_subs 2024-03-25T18:03:51.991 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.989+0000 7f41f8674640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:03:51.991 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.989+0000 7f41f8674640 1 -- 172.21.15.193:0/3656875532 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f41f0143180 con 0x7f41f00aafa0 2024-03-25T18:03:51.991 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.989+0000 7f41f8674640 1 librados: init done 2024-03-25T18:03:51.991 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.990+0000 7f41f8674640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:51.991 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.990+0000 7f41f8674640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:51.991 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.990+0000 7f41f8674640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:03:51.991 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:51.990+0000 7f41f8674640 1 -- 172.21.15.193:0/3656875532 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41f0139a00 con 0x7f41f00aafa0 2024-03-25T18:03:52.012 INFO:tasks.ceph.mgr.y.smithi066.stderr:2024-03-25T18:03:52.012+0000 7fc1b688f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:03:52.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.084+0000 7f41e6ffd640 1 -- 172.21.15.193:0/3656875532 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41e0001c50 con 0x7f41f00aafa0 2024-03-25T18:03:52.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.084+0000 7f41e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:52.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.084+0000 7f41e6ffd640 1 -- 172.21.15.193:0/3656875532 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f41e00038c0 con 0x7f41f00aafa0 2024-03-25T18:03:52.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.084+0000 7f41e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:52.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.084+0000 7f41e6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:03:52.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.084+0000 7f41e6ffd640 10 monclient: dump: 2024-03-25T18:03:52.085 INFO:teuthology.orchestra.run.smithi193.stderr:epoch 1 2024-03-25T18:03:52.085 INFO:teuthology.orchestra.run.smithi193.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:52.085 INFO:teuthology.orchestra.run.smithi193.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:52.085 INFO:teuthology.orchestra.run.smithi193.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:52.085 INFO:teuthology.orchestra.run.smithi193.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:52.085 INFO:teuthology.orchestra.run.smithi193.stderr:election_strategy: 3 2024-03-25T18:03:52.086 INFO:teuthology.orchestra.run.smithi193.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:52.086 INFO:teuthology.orchestra.run.smithi193.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:52.086 INFO:teuthology.orchestra.run.smithi193.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:52.086 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:03:52.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.084+0000 7f41e6ffd640 1 -- 172.21.15.193:0/3656875532 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f41e0003e70 con 0x7f41f00aafa0 2024-03-25T18:03:52.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.084+0000 7f41f4c0d640 4 set_mon_vals no callback set 2024-03-25T18:03:52.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.084+0000 7f41e6ffd640 1 -- 172.21.15.193:0/3656875532 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 3909999427 0 0) 0x7f41e00067b0 con 0x7f41f00aafa0 2024-03-25T18:03:52.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.084+0000 7f41e6ffd640 10 client.4156.objecter ms_dispatch 0x7f41f0000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T18:03:52.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.084+0000 7f41e6ffd640 3 client.4156.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T18:03:52.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.084+0000 7f41e6ffd640 3 client.4156.objecter handle_osd_map decoding full epoch 9 2024-03-25T18:03:52.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.084+0000 7f41e6ffd640 20 client.4156.objecter dump_active .. 0 homeless 2024-03-25T18:03:52.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.084+0000 7f41e6ffd640 1 -- 172.21.15.193:0/3656875532 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f41e0006b50 con 0x7f41f00aafa0 2024-03-25T18:03:52.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.084+0000 7f41e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:52.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.084+0000 7f41e6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:03:52.134 INFO:tasks.ceph.mgr.y.smithi066.stderr:2024-03-25T18:03:52.134+0000 7fc1b688f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:03:52.155 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.155+0000 7f41f8674640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "59a326e8-ae37-42fc-a22c-24b55a726954", "id": 7}] 2024-03-25T18:03:52.155 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.155+0000 7f41f8674640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "59a326e8-ae37-42fc-a22c-24b55a726954", "id": 7}] 2024-03-25T18:03:52.155 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.155+0000 7f41f8674640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:03:52.155 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.155+0000 7f41f8674640 1 -- 172.21.15.193:0/3656875532 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd new", "uuid": "59a326e8-ae37-42fc-a22c-24b55a726954", "id": 7} v 0) v1 -- 0x7f41f0143430 con 0x7f41f00aafa0 2024-03-25T18:03:52.284 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.285+0000 7f41e6ffd640 1 -- 172.21.15.193:0/3656875532 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 3937595769 0 0) 0x7f41e0004120 con 0x7f41f00aafa0 2024-03-25T18:03:52.518 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.518+0000 7f41e6ffd640 1 -- 172.21.15.193:0/3656875532 <== mon.1 v1:172.21.15.193:6789/0 11 ==== mon_command_ack([{"prefix": "osd new", "uuid": "59a326e8-ae37-42fc-a22c-24b55a726954", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 1005549925 0 1371665406) 0x7f41e0015d00 con 0x7f41f00aafa0 2024-03-25T18:03:52.518 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.518+0000 7f41e6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "59a326e8-ae37-42fc-a22c-24b55a726954", "id": 7}] 2024-03-25T18:03:52.518 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.518+0000 7f41e6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:03:52.519 INFO:teuthology.orchestra.run.smithi193.stdout:7 2024-03-25T18:03:52.520 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.519+0000 7f41f8674640 10 librados: watch_flush enter 2024-03-25T18:03:52.520 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.519+0000 7f41f8674640 10 librados: watch_flush exit 2024-03-25T18:03:52.520 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.519+0000 7f41f8674640 20 client.4156.objecter shutdown clearing up homeless session... 2024-03-25T18:03:52.520 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.519+0000 7f41f8674640 10 client.4156.objecter successfully canceled tick 2024-03-25T18:03:52.520 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.519+0000 7f41f8674640 10 monclient: shutdown 2024-03-25T18:03:52.520 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.519+0000 7f41f8674640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:52.520 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.519+0000 7f41f8674640 1 -- 172.21.15.193:0/3656875532 >> v1:172.21.15.193:6789/0 conn(0x7f41f00aafa0 legacy=0x7f41f01d7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:52.520 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.519+0000 7f41f8674640 1 -- 172.21.15.193:0/3656875532 shutdown_connections 2024-03-25T18:03:52.520 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.519+0000 7f41f8674640 1 -- 172.21.15.193:0/3656875532 >> 172.21.15.193:0/3656875532 conn(0x7f41f006b560 msgr2=0x7f41f0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:52.520 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.519+0000 7f41f8674640 1 -- 172.21.15.193:0/3656875532 shutdown_connections 2024-03-25T18:03:52.520 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.519+0000 7f41f8674640 1 -- 172.21.15.193:0/3656875532 wait complete. 2024-03-25T18:03:52.520 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-25T18:03:52.519+0000 7f41f8674640 1 librados: shutdown 2024-03-25T18:03:52.529 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:03:52.529 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T18:03:52.529 DEBUG:teuthology.orchestra.run.smithi066:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T18:03:52.531 INFO:tasks.ceph.osd.0:Started 2024-03-25T18:03:52.532 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:03:52.532 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T18:03:52.532 DEBUG:teuthology.orchestra.run.smithi066:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T18:03:52.534 INFO:tasks.ceph.osd.1:Started 2024-03-25T18:03:52.534 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:03:52.534 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T18:03:52.534 DEBUG:teuthology.orchestra.run.smithi066:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T18:03:52.536 INFO:tasks.ceph.osd.2:Started 2024-03-25T18:03:52.536 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:03:52.537 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T18:03:52.537 DEBUG:teuthology.orchestra.run.smithi066:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T18:03:52.539 INFO:tasks.ceph.osd.3:Started 2024-03-25T18:03:52.539 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:03:52.539 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T18:03:52.539 DEBUG:teuthology.orchestra.run.smithi193:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T18:03:52.542 INFO:tasks.ceph.osd.4:Started 2024-03-25T18:03:52.542 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:03:52.542 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T18:03:52.542 DEBUG:teuthology.orchestra.run.smithi193:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T18:03:52.544 INFO:tasks.ceph.osd.5:Started 2024-03-25T18:03:52.544 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:03:52.544 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T18:03:52.544 DEBUG:teuthology.orchestra.run.smithi193:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T18:03:52.546 INFO:tasks.ceph.osd.6:Started 2024-03-25T18:03:52.546 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:03:52.547 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T18:03:52.547 DEBUG:teuthology.orchestra.run.smithi193:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T18:03:52.549 INFO:tasks.ceph.osd.7:Started 2024-03-25T18:03:52.549 DEBUG:teuthology.orchestra.run.smithi066:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T18:03:52.723 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.723+0000 7f80aacf6640 10 monclient: get_monmap_and_config 2024-03-25T18:03:52.723 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.723+0000 7f80aacf6640 10 monclient: build_initial_monmap 2024-03-25T18:03:52.723 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.723+0000 7f80aacf6640 10 monclient: monmap: 2024-03-25T18:03:52.723 INFO:teuthology.orchestra.run.smithi066.stderr:epoch 0 2024-03-25T18:03:52.723 INFO:teuthology.orchestra.run.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:52.724 INFO:teuthology.orchestra.run.smithi066.stderr:last_changed 2024-03-25T18:03:52.723710+0000 2024-03-25T18:03:52.724 INFO:teuthology.orchestra.run.smithi066.stderr:created 2024-03-25T18:03:52.723710+0000 2024-03-25T18:03:52.724 INFO:teuthology.orchestra.run.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:52.724 INFO:teuthology.orchestra.run.smithi066.stderr:election_strategy: 3 2024-03-25T18:03:52.724 INFO:teuthology.orchestra.run.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:03:52.724 INFO:teuthology.orchestra.run.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:03:52.724 INFO:teuthology.orchestra.run.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:03:52.724 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-03-25T18:03:52.726 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.724+0000 7f80aacf6640 1 Processor -- start 2024-03-25T18:03:52.726 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.724+0000 7f80aacf6640 1 -- start start 2024-03-25T18:03:52.726 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.724+0000 7f80aacf6640 10 monclient: init 2024-03-25T18:03:52.726 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.725+0000 7f80aacf6640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:52.726 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.725+0000 7f80aacf6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:03:52.726 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.725+0000 7f80aacf6640 10 monclient(hunting): picked mon.noname-c con 0x7f80a40b4700 addr v1:172.21.15.193:6789/0 2024-03-25T18:03:52.726 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.725+0000 7f80aacf6640 10 monclient(hunting): picked mon.noname-a con 0x7f80a4142a20 addr v1:172.21.15.66:6789/0 2024-03-25T18:03:52.726 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.725+0000 7f80aacf6640 10 monclient(hunting): picked mon.noname-b con 0x7f80a40a96b0 addr v1:172.21.15.66:6790/0 2024-03-25T18:03:52.726 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.725+0000 7f80aacf6640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a40a9d60 con 0x7f80a4142a20 2024-03-25T18:03:52.726 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.725+0000 7f80aacf6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a413ff10 con 0x7f80a40b4700 2024-03-25T18:03:52.726 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.725+0000 7f80aacf6640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a40aa1f0 con 0x7f80a40a96b0 2024-03-25T18:03:52.726 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.726+0000 7f80aacf6640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:52.727 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.726+0000 7f80aacf6640 10 monclient(hunting): authenticate will time out at 6981.785180s 2024-03-25T18:03:52.727 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.726+0000 7f80a9293640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f80a40a96b0 0x7f80a413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:56096/0 (socket says 172.21.15.66:56096) 2024-03-25T18:03:52.727 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.726+0000 7f80a9293640 1 -- 172.21.15.66:0/4069227986 learned_addr learned my addr 172.21.15.66:0/4069227986 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:03:52.847 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.847+0000 7f80a37fe640 1 -- 172.21.15.66:0/4069227986 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3240016044 0 0) 0x7f80a413ff10 con 0x7f80a40b4700 2024-03-25T18:03:52.847 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.847+0000 7f80a37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:03:52.847 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.847+0000 7f80a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:52.847 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.847+0000 7f80a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:52.847 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.847+0000 7f80a37fe640 10 monclient(hunting): my global_id is 4168 2024-03-25T18:03:52.847 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.847+0000 7f80a37fe640 1 -- 172.21.15.66:0/4069227986 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8094003520 con 0x7f80a40b4700 2024-03-25T18:03:52.852 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.852+0000 7f80a37fe640 1 -- 172.21.15.66:0/4069227986 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2301405706 0 0) 0x7f8094003520 con 0x7f80a40b4700 2024-03-25T18:03:52.852 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.852+0000 7f80a37fe640 1 -- 172.21.15.66:0/4069227986 >> v1:172.21.15.66:6790/0 conn(0x7f80a40a96b0 legacy=0x7f80a413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:52.853 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.852+0000 7f80a37fe640 1 -- 172.21.15.66:0/4069227986 >> v1:172.21.15.66:6789/0 conn(0x7f80a4142a20 legacy=0x7f80a413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:52.853 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.852+0000 7f80a37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:03:52.853 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.852+0000 7f80a37fe640 1 monclient: found mon.noname-c 2024-03-25T18:03:52.853 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.852+0000 7f80a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:52.853 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.852+0000 7f80a37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:03:52.853 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.852+0000 7f80a37fe640 1 -- 172.21.15.66:0/4069227986 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a40b4ae0 con 0x7f80a40b4700 2024-03-25T18:03:52.853 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.852+0000 7f80a37fe640 10 monclient: _finish_auth 0 2024-03-25T18:03:52.853 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.852+0000 7f80a37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:03:52.853 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.852+0000 7f80a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:52.853 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.852+0000 7f80aacf6640 5 monclient: authenticate success, global_id 4168 2024-03-25T18:03:52.853 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.852+0000 7f80aacf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:52.854 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.853+0000 7f80a37fe640 1 -- 172.21.15.66:0/4069227986 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8098001690 con 0x7f80a40b4700 2024-03-25T18:03:52.854 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.853+0000 7f80a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:52.854 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.853+0000 7f80a37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:03:52.854 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.853+0000 7f80a37fe640 10 monclient: dump: 2024-03-25T18:03:52.854 INFO:teuthology.orchestra.run.smithi066.stderr:epoch 1 2024-03-25T18:03:52.854 INFO:teuthology.orchestra.run.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:52.854 INFO:teuthology.orchestra.run.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:52.854 INFO:teuthology.orchestra.run.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:52.854 INFO:teuthology.orchestra.run.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:52.854 INFO:teuthology.orchestra.run.smithi066.stderr:election_strategy: 3 2024-03-25T18:03:52.854 INFO:teuthology.orchestra.run.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:52.854 INFO:teuthology.orchestra.run.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:52.854 INFO:teuthology.orchestra.run.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:52.854 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-03-25T18:03:52.855 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.853+0000 7f80aacf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:52.855 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.855+0000 7f80a37fe640 1 -- 172.21.15.66:0/4069227986 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8098001920 con 0x7f80a40b4700 2024-03-25T18:03:52.855 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.855+0000 7f80a37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:52.855 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.855+0000 7f80a37fe640 1 -- 172.21.15.66:0/4069227986 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8098001690 con 0x7f80a40b4700 2024-03-25T18:03:52.855 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.855+0000 7f80aacf6640 10 monclient: get_monmap_and_config success 2024-03-25T18:03:52.855 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.855+0000 7f80aacf6640 4 set_mon_vals no callback set 2024-03-25T18:03:52.855 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.855+0000 7f80a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:52.855 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.855+0000 7f80a37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:03:52.855 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.855+0000 7f80a37fe640 10 monclient: dump: 2024-03-25T18:03:52.855 INFO:teuthology.orchestra.run.smithi066.stderr:epoch 1 2024-03-25T18:03:52.856 INFO:teuthology.orchestra.run.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:52.856 INFO:teuthology.orchestra.run.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:52.856 INFO:teuthology.orchestra.run.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:52.856 INFO:teuthology.orchestra.run.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:52.856 INFO:teuthology.orchestra.run.smithi066.stderr:election_strategy: 3 2024-03-25T18:03:52.856 INFO:teuthology.orchestra.run.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:52.856 INFO:teuthology.orchestra.run.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:52.856 INFO:teuthology.orchestra.run.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:52.856 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-03-25T18:03:52.857 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.856+0000 7f80aacf6640 10 monclient: shutdown 2024-03-25T18:03:52.857 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.856+0000 7f80aacf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:52.857 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.856+0000 7f80aacf6640 1 -- 172.21.15.66:0/4069227986 >> v1:172.21.15.193:6789/0 conn(0x7f80a40b4700 legacy=0x7f80a4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:52.857 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.856+0000 7f80aacf6640 1 -- 172.21.15.66:0/4069227986 shutdown_connections 2024-03-25T18:03:52.857 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.856+0000 7f80aacf6640 1 -- 172.21.15.66:0/4069227986 >> 172.21.15.66:0/4069227986 conn(0x7f80a40668e0 msgr2=0x7f80a4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:52.857 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.856+0000 7f80aacf6640 1 -- 172.21.15.66:0/4069227986 shutdown_connections 2024-03-25T18:03:52.857 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.856+0000 7f80aacf6640 1 -- 172.21.15.66:0/4069227986 wait complete. 2024-03-25T18:03:52.858 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.857+0000 7f80aacf6640 10 monclient: build_initial_monmap 2024-03-25T18:03:52.858 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.857+0000 7f80aacf6640 10 monclient: monmap: 2024-03-25T18:03:52.858 INFO:teuthology.orchestra.run.smithi066.stderr:epoch 0 2024-03-25T18:03:52.858 INFO:teuthology.orchestra.run.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:03:52.858 INFO:teuthology.orchestra.run.smithi066.stderr:last_changed 0.000000 2024-03-25T18:03:52.859 INFO:teuthology.orchestra.run.smithi066.stderr:created 0.000000 2024-03-25T18:03:52.859 INFO:teuthology.orchestra.run.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:52.859 INFO:teuthology.orchestra.run.smithi066.stderr:election_strategy: 1 2024-03-25T18:03:52.859 INFO:teuthology.orchestra.run.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:03:52.859 INFO:teuthology.orchestra.run.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:03:52.859 INFO:teuthology.orchestra.run.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:03:52.859 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-03-25T18:03:52.859 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.858+0000 7f80aacf6640 1 librados: starting msgr at 2024-03-25T18:03:52.859 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.858+0000 7f80aacf6640 1 librados: starting objecter 2024-03-25T18:03:52.859 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.858+0000 7f80aacf6640 1 Processor -- start 2024-03-25T18:03:52.859 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.858+0000 7f80aacf6640 1 -- start start 2024-03-25T18:03:52.860 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.858+0000 7f80aacf6640 1 librados: setting wanted keys 2024-03-25T18:03:52.860 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.858+0000 7f80aacf6640 1 librados: calling monclient init 2024-03-25T18:03:52.860 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.858+0000 7f80aacf6640 10 monclient: init 2024-03-25T18:03:52.861 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.859+0000 7f80aacf6640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:52.861 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.859+0000 7f80aacf6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:03:52.861 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.859+0000 7f80aacf6640 10 monclient(hunting): picked mon.noname-c con 0x7f80a40a96b0 addr v1:172.21.15.66:6790/0 2024-03-25T18:03:52.861 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.859+0000 7f80aacf6640 10 monclient(hunting): picked mon.noname-a con 0x7f80a40b4700 addr v1:172.21.15.66:6789/0 2024-03-25T18:03:52.861 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.860+0000 7f80aacf6640 10 monclient(hunting): picked mon.noname-b con 0x7f80a4142a20 addr v1:172.21.15.193:6789/0 2024-03-25T18:03:52.862 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.860+0000 7f80aacf6640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a4137520 con 0x7f80a40b4700 2024-03-25T18:03:52.862 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.860+0000 7f80aacf6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a4137800 con 0x7f80a4142a20 2024-03-25T18:03:52.862 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.860+0000 7f80aacf6640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a4137ae0 con 0x7f80a40a96b0 2024-03-25T18:03:52.862 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.860+0000 7f80aacf6640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:52.862 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.860+0000 7f80aacf6640 10 monclient(hunting): authenticate will time out at 6981.919514s 2024-03-25T18:03:52.862 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.860+0000 7f80a9293640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f80a4142a20 0x7f80a41381e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:51626/0 (socket says 172.21.15.66:51626) 2024-03-25T18:03:52.862 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.860+0000 7f80a9293640 1 -- 172.21.15.66:0/910865655 learned_addr learned my addr 172.21.15.66:0/910865655 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:03:52.862 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.861+0000 7f80a17fa640 10 client.?.objecter ms_handle_connect 0x7f80a4142a20 2024-03-25T18:03:52.862 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.861+0000 7f80a17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:52.865 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.864+0000 7f80a17fa640 1 -- 172.21.15.66:0/910865655 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 554733261 0 0) 0x7f80a4137800 con 0x7f80a4142a20 2024-03-25T18:03:52.865 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.864+0000 7f80a17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:03:52.865 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.864+0000 7f80a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:52.865 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.864+0000 7f80a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:52.865 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.864+0000 7f80a17fa640 10 monclient(hunting): my global_id is 4171 2024-03-25T18:03:52.865 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.864+0000 7f80a17fa640 1 -- 172.21.15.66:0/910865655 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8088004eb0 con 0x7f80a4142a20 2024-03-25T18:03:52.865 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.865+0000 7f80a17fa640 10 client.?.objecter ms_handle_connect 0x7f80a40a96b0 2024-03-25T18:03:52.865 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.865+0000 7f80a17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:52.869 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.869+0000 7f80a17fa640 1 -- 172.21.15.66:0/910865655 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130309873 0 0) 0x7f80a4137ae0 con 0x7f80a40a96b0 2024-03-25T18:03:52.869 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.869+0000 7f80a17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:03:52.869 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.869+0000 7f80a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:52.869 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.869+0000 7f80a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:52.869 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.869+0000 7f80a17fa640 10 monclient(hunting): my global_id is 4163 2024-03-25T18:03:52.869 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.869+0000 7f80a17fa640 1 -- 172.21.15.66:0/910865655 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80a4137800 con 0x7f80a40a96b0 2024-03-25T18:03:52.873 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.872+0000 7f80a17fa640 1 -- 172.21.15.66:0/910865655 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2072156952 0 0) 0x7f8088004eb0 con 0x7f80a4142a20 2024-03-25T18:03:52.873 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.872+0000 7f80a17fa640 1 -- 172.21.15.66:0/910865655 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80a4137ae0 con 0x7f80a4142a20 2024-03-25T18:03:52.873 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.872+0000 7f80a17fa640 1 -- 172.21.15.66:0/910865655 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f809c001bb0 con 0x7f80a4142a20 2024-03-25T18:03:52.873 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.872+0000 7f80a17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:52.873 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.872+0000 7f80a17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:03:52.873 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.872+0000 7f80a17fa640 10 monclient(hunting): dump: 2024-03-25T18:03:52.873 INFO:teuthology.orchestra.run.smithi066.stderr:epoch 1 2024-03-25T18:03:52.873 INFO:teuthology.orchestra.run.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:52.873 INFO:teuthology.orchestra.run.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:52.874 INFO:teuthology.orchestra.run.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:52.874 INFO:teuthology.orchestra.run.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:52.874 INFO:teuthology.orchestra.run.smithi066.stderr:election_strategy: 3 2024-03-25T18:03:52.874 INFO:teuthology.orchestra.run.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:52.874 INFO:teuthology.orchestra.run.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:52.874 INFO:teuthology.orchestra.run.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:52.874 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-03-25T18:03:52.874 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.872+0000 7f80a17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:03:52.874 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.872+0000 7f80a17fa640 10 client.?.objecter ms_handle_connect 0x7f80a40b4700 2024-03-25T18:03:52.874 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.872+0000 7f80a17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:52.877 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.876+0000 7f80a17fa640 1 -- 172.21.15.66:0/910865655 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 506164759 0 0) 0x7f80a4137800 con 0x7f80a40a96b0 2024-03-25T18:03:52.877 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.877+0000 7f80a17fa640 1 -- 172.21.15.66:0/910865655 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8088004eb0 con 0x7f80a40a96b0 2024-03-25T18:03:52.877 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.877+0000 7f80a17fa640 1 -- 172.21.15.66:0/910865655 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8098001c70 con 0x7f80a40a96b0 2024-03-25T18:03:52.877 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.877+0000 7f80a17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:52.877 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.877+0000 7f80a17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:03:52.877 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.877+0000 7f80a17fa640 10 monclient(hunting): dump: 2024-03-25T18:03:52.877 INFO:teuthology.orchestra.run.smithi066.stderr:epoch 1 2024-03-25T18:03:52.878 INFO:teuthology.orchestra.run.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:52.878 INFO:teuthology.orchestra.run.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:52.878 INFO:teuthology.orchestra.run.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:52.878 INFO:teuthology.orchestra.run.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:52.878 INFO:teuthology.orchestra.run.smithi066.stderr:election_strategy: 3 2024-03-25T18:03:52.878 INFO:teuthology.orchestra.run.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:52.878 INFO:teuthology.orchestra.run.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:52.878 INFO:teuthology.orchestra.run.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:52.878 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-03-25T18:03:52.880 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.877+0000 7f80a17fa640 1 -- 172.21.15.66:0/910865655 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2965479161 0 0) 0x7f80a4137ae0 con 0x7f80a4142a20 2024-03-25T18:03:52.880 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.877+0000 7f80a17fa640 1 -- 172.21.15.66:0/910865655 >> v1:172.21.15.66:6790/0 conn(0x7f80a40a96b0 legacy=0x7f80a4134180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:52.880 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.877+0000 7f80a17fa640 1 -- 172.21.15.66:0/910865655 >> v1:172.21.15.66:6789/0 conn(0x7f80a40b4700 legacy=0x7f80a4135f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:52.881 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.877+0000 7f80a17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:03:52.881 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.877+0000 7f80a17fa640 1 monclient: found mon.b 2024-03-25T18:03:52.881 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.878+0000 7f80a17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:52.881 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.878+0000 7f80a17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:03:52.881 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.878+0000 7f80a17fa640 1 -- 172.21.15.66:0/910865655 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a4137dc0 con 0x7f80a4142a20 2024-03-25T18:03:52.881 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.878+0000 7f80a17fa640 10 monclient: _finish_auth 0 2024-03-25T18:03:52.881 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.878+0000 7f80a17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:03:52.881 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.878+0000 7f80a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:52.881 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.878+0000 7f80aacf6640 5 monclient: authenticate success, global_id 4171 2024-03-25T18:03:52.881 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.878+0000 7f80aacf6640 10 monclient: _renew_subs 2024-03-25T18:03:52.881 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.878+0000 7f80aacf6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:03:52.881 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.878+0000 7f80aacf6640 1 -- 172.21.15.66:0/910865655 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a41e34b0 con 0x7f80a4142a20 2024-03-25T18:03:52.881 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.878+0000 7f80aacf6640 10 client.4171.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:03:52.881 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.878+0000 7f80aacf6640 10 monclient: _renew_subs 2024-03-25T18:03:52.881 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.879+0000 7f80aacf6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:03:52.881 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.879+0000 7f80aacf6640 1 -- 172.21.15.66:0/910865655 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a41e3820 con 0x7f80a4142a20 2024-03-25T18:03:52.882 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.879+0000 7f80aacf6640 1 librados: init done 2024-03-25T18:03:52.882 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.880+0000 7f80aacf6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:52.882 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.880+0000 7f80aacf6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:52.882 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.880+0000 7f80aacf6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:03:52.882 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:52.880+0000 7f80aacf6640 1 -- 172.21.15.66:0/910865655 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80a4136870 con 0x7f80a4142a20 2024-03-25T18:03:53.065 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.065+0000 7f80a17fa640 1 -- 172.21.15.66:0/910865655 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f809c0020a0 con 0x7f80a4142a20 2024-03-25T18:03:53.065 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.065+0000 7f80a17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:53.065 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.065+0000 7f80a17fa640 1 -- 172.21.15.66:0/910865655 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f809c004d00 con 0x7f80a4142a20 2024-03-25T18:03:53.066 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.065+0000 7f80a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:53.066 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.065+0000 7f80a17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:03:53.066 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.065+0000 7f80a17fa640 10 monclient: dump: 2024-03-25T18:03:53.066 INFO:teuthology.orchestra.run.smithi066.stderr:epoch 1 2024-03-25T18:03:53.066 INFO:teuthology.orchestra.run.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:53.066 INFO:teuthology.orchestra.run.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:53.066 INFO:teuthology.orchestra.run.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:53.066 INFO:teuthology.orchestra.run.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:53.066 INFO:teuthology.orchestra.run.smithi066.stderr:election_strategy: 3 2024-03-25T18:03:53.066 INFO:teuthology.orchestra.run.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:53.067 INFO:teuthology.orchestra.run.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:53.067 INFO:teuthology.orchestra.run.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:53.067 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-03-25T18:03:53.067 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.065+0000 7f80a2ffd640 4 set_mon_vals no callback set 2024-03-25T18:03:53.067 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.065+0000 7f80a17fa640 1 -- 172.21.15.66:0/910865655 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 3937595769 0 0) 0x7f809c004f20 con 0x7f80a4142a20 2024-03-25T18:03:53.069 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.069+0000 7f80a17fa640 1 -- 172.21.15.66:0/910865655 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f809c015a10 con 0x7f80a4142a20 2024-03-25T18:03:53.069 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.069+0000 7f80a17fa640 10 client.4171.objecter ms_dispatch 0x7f80a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:03:53.069 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.069+0000 7f80a17fa640 3 client.4171.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:03:53.070 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.069+0000 7f80a17fa640 3 client.4171.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:03:53.070 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.069+0000 7f80a17fa640 20 client.4171.objecter dump_active .. 0 homeless 2024-03-25T18:03:53.070 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.069+0000 7f80a17fa640 1 -- 172.21.15.66:0/910865655 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f809c014cd0 con 0x7f80a4142a20 2024-03-25T18:03:53.070 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.069+0000 7f80a17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:53.070 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.069+0000 7f80a17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:03:53.193 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.193+0000 7f80aacf6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:03:53.194 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.193+0000 7f80aacf6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:03:53.194 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.193+0000 7f80aacf6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:03:53.194 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.193+0000 7f80aacf6640 1 -- 172.21.15.66:0/910865655 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80a404f730 con 0x7f80a4142a20 2024-03-25T18:03:53.274 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.274+0000 7f80a17fa640 1 -- 172.21.15.66:0/910865655 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f809c0134b0 con 0x7f80a4142a20 2024-03-25T18:03:53.274 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.274+0000 7f80a17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:03:53.274 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.274+0000 7f80a17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:03:53.275 INFO:teuthology.orchestra.run.smithi066.stdout: 2024-03-25T18:03:53.275 INFO:teuthology.orchestra.run.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:03:53.276 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.275+0000 7f80aacf6640 10 librados: watch_flush enter 2024-03-25T18:03:53.276 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.275+0000 7f80aacf6640 10 librados: watch_flush exit 2024-03-25T18:03:53.276 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.275+0000 7f80aacf6640 20 client.4171.objecter shutdown clearing up homeless session... 2024-03-25T18:03:53.277 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.275+0000 7f80aacf6640 10 client.4171.objecter successfully canceled tick 2024-03-25T18:03:53.277 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.275+0000 7f80aacf6640 10 monclient: shutdown 2024-03-25T18:03:53.277 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.276+0000 7f80aacf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:53.277 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.276+0000 7f80aacf6640 1 -- 172.21.15.66:0/910865655 >> v1:172.21.15.193:6789/0 conn(0x7f80a4142a20 legacy=0x7f80a41381e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:53.277 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.276+0000 7f80aacf6640 1 -- 172.21.15.66:0/910865655 shutdown_connections 2024-03-25T18:03:53.277 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.276+0000 7f80aacf6640 1 -- 172.21.15.66:0/910865655 >> 172.21.15.66:0/910865655 conn(0x7f80a40668e0 msgr2=0x7f80a413f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:53.277 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.276+0000 7f80aacf6640 1 -- 172.21.15.66:0/910865655 shutdown_connections 2024-03-25T18:03:53.277 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.276+0000 7f80aacf6640 1 -- 172.21.15.66:0/910865655 wait complete. 2024-03-25T18:03:53.277 INFO:teuthology.orchestra.run.smithi066.stderr:2024-03-25T18:03:53.276+0000 7f80aacf6640 1 librados: shutdown 2024-03-25T18:03:53.291 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T18:03:53.291 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T18:03:53.663 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T18:03:53.663+0000 7f5dab43d640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T18:03:53.679 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T18:03:53.680+0000 7f5dab43d640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T18:03:54.097 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:03:54.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.255+0000 7f4764bcd640 10 monclient: get_monmap_and_config 2024-03-25T18:03:54.256 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.255+0000 7f4764bcd640 10 monclient: build_initial_monmap 2024-03-25T18:03:54.256 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.255+0000 7f4764bcd640 10 monclient: monmap: 2024-03-25T18:03:54.256 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:03:54.256 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:54.256 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:03:54.256096+0000 2024-03-25T18:03:54.256 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:03:54.256096+0000 2024-03-25T18:03:54.256 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:54.256 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:03:54.256 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:03:54.256 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:03:54.256 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:03:54.256 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:03:54.258 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.256+0000 7f4764bcd640 1 Processor -- start 2024-03-25T18:03:54.258 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.256+0000 7f4764bcd640 1 -- start start 2024-03-25T18:03:54.258 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.256+0000 7f4764bcd640 10 monclient: init 2024-03-25T18:03:54.258 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.257+0000 7f4764bcd640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:54.258 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.257+0000 7f4764bcd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:03:54.259 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.257+0000 7f4764bcd640 10 monclient(hunting): picked mon.noname-c con 0x7f47600a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:03:54.259 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.257+0000 7f4764bcd640 10 monclient(hunting): picked mon.noname-b con 0x7f476013f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:03:54.259 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.257+0000 7f4764bcd640 10 monclient(hunting): picked mon.noname-a con 0x7f47601430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:03:54.259 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.257+0000 7f4764bcd640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4760142520 con 0x7f47601430c0 2024-03-25T18:03:54.259 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.257+0000 7f4764bcd640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47600b28a0 con 0x7f47600a98f0 2024-03-25T18:03:54.259 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.257+0000 7f4764bcd640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47600b2a30 con 0x7f476013f710 2024-03-25T18:03:54.259 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.257+0000 7f4764bcd640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:54.259 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.257+0000 7f4764bcd640 10 monclient(hunting): authenticate will time out at 6983.316971s 2024-03-25T18:03:54.259 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.258+0000 7f4757fff640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f476013f710 0x7f47601429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:52556/0 (socket says 172.21.15.66:52556) 2024-03-25T18:03:54.259 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.259+0000 7f4757fff640 1 -- 172.21.15.66:0/2878991100 learned_addr learned my addr 172.21.15.66:0/2878991100 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:03:54.264 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.263+0000 7f475dd9b640 1 -- 172.21.15.66:0/2878991100 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1900733992 0 0) 0x7f47600b2a30 con 0x7f476013f710 2024-03-25T18:03:54.264 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.264+0000 7f475dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:03:54.264 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.264+0000 7f475dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:54.264 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.264+0000 7f475dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:54.264 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.264+0000 7f475dd9b640 10 monclient(hunting): my global_id is 4172 2024-03-25T18:03:54.264 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.264+0000 7f475dd9b640 1 -- 172.21.15.66:0/2878991100 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4740003520 con 0x7f476013f710 2024-03-25T18:03:54.270 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.269+0000 7f475dd9b640 1 -- 172.21.15.66:0/2878991100 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1123011587 0 0) 0x7f4740003520 con 0x7f476013f710 2024-03-25T18:03:54.270 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.269+0000 7f475dd9b640 1 -- 172.21.15.66:0/2878991100 >> v1:172.21.15.193:6789/0 conn(0x7f47600a98f0 legacy=0x7f47600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:03:54.270 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.269+0000 7f475dd9b640 1 -- 172.21.15.66:0/2878991100 >> v1:172.21.15.66:6789/0 conn(0x7f47601430c0 legacy=0x7f476014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:03:54.270 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.269+0000 7f475dd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:03:54.270 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.269+0000 7f475dd9b640 1 monclient: found mon.noname-b 2024-03-25T18:03:54.271 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.269+0000 7f475dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:54.271 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.269+0000 7f475dd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.66:6790/0 2024-03-25T18:03:54.271 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.269+0000 7f475dd9b640 1 -- 172.21.15.66:0/2878991100 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47600b44c0 con 0x7f476013f710 2024-03-25T18:03:54.271 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.269+0000 7f475dd9b640 10 monclient: _finish_auth 0 2024-03-25T18:03:54.271 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.269+0000 7f475dd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:03:54.271 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.269+0000 7f475dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:54.271 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.269+0000 7f4764bcd640 5 monclient: authenticate success, global_id 4172 2024-03-25T18:03:54.272 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.269+0000 7f4764bcd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:54.272 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.269+0000 7f475dd9b640 1 -- 172.21.15.66:0/2878991100 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f4750001670 con 0x7f476013f710 2024-03-25T18:03:54.272 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.269+0000 7f475dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:54.272 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.269+0000 7f475dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:03:54.272 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.269+0000 7f475dd9b640 10 monclient: dump: 2024-03-25T18:03:54.272 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:03:54.272 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:54.272 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:54.272 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:54.273 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:54.273 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:03:54.273 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:54.273 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:54.273 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:54.273 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:03:54.273 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.269+0000 7f4764bcd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:03:54.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.272+0000 7f475dd9b640 1 -- 172.21.15.66:0/2878991100 <== mon.2 v1:172.21.15.66:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47500018d0 con 0x7f476013f710 2024-03-25T18:03:54.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.272+0000 7f475dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:54.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.272+0000 7f4764bcd640 10 monclient: get_monmap_and_config success 2024-03-25T18:03:54.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.272+0000 7f4764bcd640 4 set_mon_vals no callback set 2024-03-25T18:03:54.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.273+0000 7f4764bcd640 10 monclient: shutdown 2024-03-25T18:03:54.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.273+0000 7f4764bcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:54.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.273+0000 7f4764bcd640 1 -- 172.21.15.66:0/2878991100 >> v1:172.21.15.66:6790/0 conn(0x7f476013f710 legacy=0x7f47601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:54.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.274+0000 7f475dd9b640 1 -- 172.21.15.66:0/2878991100 <== mon.2 v1:172.21.15.66:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f4750001670 con 0x7f476013f710 2024-03-25T18:03:54.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.274+0000 7f475dd9b640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-25T18:03:54.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.274+0000 7f4764bcd640 1 -- 172.21.15.66:0/2878991100 shutdown_connections 2024-03-25T18:03:54.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.274+0000 7f4764bcd640 1 -- 172.21.15.66:0/2878991100 >> 172.21.15.66:0/2878991100 conn(0x7f47600b7ea0 msgr2=0x7f47600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:54.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.274+0000 7f4764bcd640 1 -- 172.21.15.66:0/2878991100 shutdown_connections 2024-03-25T18:03:54.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.274+0000 7f4764bcd640 1 -- 172.21.15.66:0/2878991100 wait complete. 2024-03-25T18:03:54.278 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.275+0000 7f4764bcd640 10 monclient: build_initial_monmap 2024-03-25T18:03:54.278 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.276+0000 7f4764bcd640 10 monclient: monmap: 2024-03-25T18:03:54.278 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:03:54.278 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:03:54.278 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:03:54.278 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:03:54.279 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:03:54.279 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:03:54.279 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:03:54.279 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:03:54.279 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:03:54.279 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:03:54.279 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.276+0000 7f4764bcd640 1 librados: starting msgr at 2024-03-25T18:03:54.279 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.276+0000 7f4764bcd640 1 librados: starting objecter 2024-03-25T18:03:54.279 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.276+0000 7f4764bcd640 1 Processor -- start 2024-03-25T18:03:54.279 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.276+0000 7f4764bcd640 1 -- start start 2024-03-25T18:03:54.279 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.276+0000 7f4764bcd640 1 librados: setting wanted keys 2024-03-25T18:03:54.279 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.276+0000 7f4764bcd640 1 librados: calling monclient init 2024-03-25T18:03:54.279 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.276+0000 7f4764bcd640 10 monclient: init 2024-03-25T18:03:54.279 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.277+0000 7f4764bcd640 10 monclient: _reopen_session rank -1 2024-03-25T18:03:54.279 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.277+0000 7f4764bcd640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:03:54.279 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.277+0000 7f4764bcd640 10 monclient(hunting): picked mon.noname-a con 0x7f47600a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:03:54.280 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.277+0000 7f4764bcd640 10 monclient(hunting): picked mon.noname-b con 0x7f47601430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:03:54.280 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.277+0000 7f4764bcd640 10 monclient(hunting): picked mon.noname-c con 0x7f476013a430 addr v1:172.21.15.66:6790/0 2024-03-25T18:03:54.280 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.277+0000 7f4764bcd640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f476013bb60 con 0x7f47600a98f0 2024-03-25T18:03:54.280 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.278+0000 7f4764bcd640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47601dba70 con 0x7f47601430c0 2024-03-25T18:03:54.280 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.278+0000 7f4764bcd640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f476013b460 con 0x7f476013a430 2024-03-25T18:03:54.280 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.278+0000 7f4764bcd640 10 monclient(hunting): _renew_subs 2024-03-25T18:03:54.280 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.278+0000 7f4764bcd640 10 monclient(hunting): authenticate will time out at 6983.337412s 2024-03-25T18:03:54.280 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.278+0000 7f475e59c640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f47600a98f0 0x7f4760139d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:46752/0 (socket says 172.21.15.66:46752) 2024-03-25T18:03:54.280 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.278+0000 7f475e59c640 1 -- 172.21.15.66:0/811863265 learned_addr learned my addr 172.21.15.66:0/811863265 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:03:54.281 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.278+0000 7f4756ffd640 10 client.?.objecter ms_handle_connect 0x7f47600a98f0 2024-03-25T18:03:54.281 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.278+0000 7f4756ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:54.281 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.279+0000 7f4756ffd640 10 client.?.objecter ms_handle_connect 0x7f476013a430 2024-03-25T18:03:54.281 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.279+0000 7f4756ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:54.282 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.281+0000 7f4756ffd640 1 -- 172.21.15.66:0/811863265 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3546951619 0 0) 0x7f476013bb60 con 0x7f47600a98f0 2024-03-25T18:03:54.282 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.281+0000 7f4756ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:03:54.282 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.281+0000 7f4756ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:54.282 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.281+0000 7f4756ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:54.282 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.281+0000 7f4756ffd640 10 monclient(hunting): my global_id is 4176 2024-03-25T18:03:54.282 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.281+0000 7f4756ffd640 1 -- 172.21.15.66:0/811863265 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4744004eb0 con 0x7f47600a98f0 2024-03-25T18:03:54.283 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.282+0000 7f4756ffd640 1 -- 172.21.15.66:0/811863265 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1457682400 0 0) 0x7f476013b460 con 0x7f476013a430 2024-03-25T18:03:54.283 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.282+0000 7f4756ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:03:54.283 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.282+0000 7f4756ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:03:54.283 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.282+0000 7f4756ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:03:54.283 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.282+0000 7f4756ffd640 10 monclient(hunting): my global_id is 4175 2024-03-25T18:03:54.283 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.282+0000 7f4756ffd640 1 -- 172.21.15.66:0/811863265 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f476013bb60 con 0x7f476013a430 2024-03-25T18:03:54.290 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.290+0000 7f4756ffd640 1 -- 172.21.15.66:0/811863265 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2900948555 0 0) 0x7f476013bb60 con 0x7f476013a430 2024-03-25T18:03:54.290 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.290+0000 7f4756ffd640 1 -- 172.21.15.66:0/811863265 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f476013b460 con 0x7f476013a430 2024-03-25T18:03:54.290 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.290+0000 7f4756ffd640 1 -- 172.21.15.66:0/811863265 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f4758001a90 con 0x7f476013a430 2024-03-25T18:03:54.290 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.290+0000 7f4756ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:54.291 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.290+0000 7f4756ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:03:54.291 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.290+0000 7f4756ffd640 10 monclient(hunting): dump: 2024-03-25T18:03:54.291 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:03:54.291 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:54.291 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:54.291 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:54.291 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:54.291 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:03:54.291 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:54.291 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:54.291 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:54.291 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:03:54.291 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.290+0000 7f4756ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:03:54.291 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.290+0000 7f4756ffd640 1 -- 172.21.15.66:0/811863265 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1775594970 0 0) 0x7f4744004eb0 con 0x7f47600a98f0 2024-03-25T18:03:54.292 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.290+0000 7f4756ffd640 1 -- 172.21.15.66:0/811863265 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f476013bb60 con 0x7f47600a98f0 2024-03-25T18:03:54.292 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.290+0000 7f4756ffd640 1 -- 172.21.15.66:0/811863265 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f47480018e0 con 0x7f47600a98f0 2024-03-25T18:03:54.292 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.290+0000 7f4756ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:54.292 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.290+0000 7f4756ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:03:54.292 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.290+0000 7f4756ffd640 10 monclient(hunting): dump: 2024-03-25T18:03:54.292 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:03:54.292 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:54.292 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:54.292 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:54.292 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:54.292 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:03:54.292 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:54.292 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:54.292 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:54.292 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:03:54.293 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.293+0000 7f4756ffd640 10 client.?.objecter ms_handle_connect 0x7f47601430c0 2024-03-25T18:03:54.294 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.293+0000 7f4756ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:03:54.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.296+0000 7f4756ffd640 1 -- 172.21.15.66:0/811863265 <== mon.2 v1:172.21.15.66:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2393012867 0 0) 0x7f476013b460 con 0x7f476013a430 2024-03-25T18:03:54.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.296+0000 7f4756ffd640 1 -- 172.21.15.66:0/811863265 >> v1:172.21.15.193:6789/0 conn(0x7f47601430c0 legacy=0x7f47601dac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:54.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.296+0000 7f4756ffd640 1 -- 172.21.15.66:0/811863265 >> v1:172.21.15.66:6789/0 conn(0x7f47600a98f0 legacy=0x7f4760139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:54.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.296+0000 7f4756ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:03:54.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.296+0000 7f4756ffd640 1 monclient: found mon.c 2024-03-25T18:03:54.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.296+0000 7f4756ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:03:54.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.296+0000 7f4756ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:03:54.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.296+0000 7f4756ffd640 1 -- 172.21.15.66:0/811863265 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f476013b740 con 0x7f476013a430 2024-03-25T18:03:54.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.296+0000 7f4756ffd640 10 monclient: _finish_auth 0 2024-03-25T18:03:54.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.296+0000 7f4756ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:03:54.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.296+0000 7f4756ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:03:54.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.296+0000 7f4764bcd640 5 monclient: authenticate success, global_id 4175 2024-03-25T18:03:54.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.296+0000 7f4764bcd640 10 monclient: _renew_subs 2024-03-25T18:03:54.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.296+0000 7f4764bcd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:03:54.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.296+0000 7f4764bcd640 1 -- 172.21.15.66:0/811863265 --> v1:172.21.15.66:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f476013ef00 con 0x7f476013a430 2024-03-25T18:03:54.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.296+0000 7f4764bcd640 10 client.4175.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:03:54.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.296+0000 7f4764bcd640 10 monclient: _renew_subs 2024-03-25T18:03:54.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.296+0000 7f4764bcd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:03:54.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.296+0000 7f4764bcd640 1 -- 172.21.15.66:0/811863265 --> v1:172.21.15.66:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4760139920 con 0x7f476013a430 2024-03-25T18:03:54.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.296+0000 7f4764bcd640 1 librados: init done 2024-03-25T18:03:54.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.298+0000 7f4764bcd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:54.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.298+0000 7f4764bcd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:54.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.298+0000 7f4764bcd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:03:54.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.298+0000 7f4764bcd640 1 -- 172.21.15.66:0/811863265 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47600627b0 con 0x7f476013a430 2024-03-25T18:03:54.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.339+0000 7f4756ffd640 1 -- 172.21.15.66:0/811863265 <== mon.2 v1:172.21.15.66:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4758001fa0 con 0x7f476013a430 2024-03-25T18:03:54.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.339+0000 7f4756ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:03:54.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.339+0000 7f4756ffd640 1 -- 172.21.15.66:0/811863265 <== mon.2 v1:172.21.15.66:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f4758003b10 con 0x7f476013a430 2024-03-25T18:03:54.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.339+0000 7f4756ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:03:54.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.339+0000 7f4756ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:03:54.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.339+0000 7f4756ffd640 10 monclient: dump: 2024-03-25T18:03:54.340 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:03:54.340 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:03:54.340 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:54.340 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:03:54.340 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:03:54.340 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:03:54.340 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:03:54.340 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:03:54.340 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:03:54.340 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:03:54.341 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.339+0000 7f475d59a640 4 set_mon_vals no callback set 2024-03-25T18:03:54.341 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.340+0000 7f4756ffd640 1 -- 172.21.15.66:0/811863265 <== mon.2 v1:172.21.15.66:6790/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 2364263485 0 0) 0x7f4758003d30 con 0x7f476013a430 2024-03-25T18:03:54.341 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.341+0000 7f475e59c640 10 monclient: get_auth_request con 0x7f474406dd60 auth_method 0 2024-03-25T18:03:54.341 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.341+0000 7f4756ffd640 10 client.4175.objecter ms_handle_connect 0x7f474406dd60 2024-03-25T18:03:54.344 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.344+0000 7f4756ffd640 1 -- 172.21.15.66:0/811863265 <== mon.2 v1:172.21.15.66:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f4758053720 con 0x7f476013a430 2024-03-25T18:03:54.344 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.344+0000 7f4756ffd640 10 client.4175.objecter ms_dispatch 0x7f4760000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:03:54.344 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.344+0000 7f4756ffd640 3 client.4175.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:03:54.344 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.344+0000 7f4756ffd640 3 client.4175.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:03:54.344 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.344+0000 7f4756ffd640 20 client.4175.objecter dump_active .. 0 homeless 2024-03-25T18:03:54.787 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.787+0000 7f4756ffd640 1 -- 172.21.15.66:0/811863265 <== mon.2 v1:172.21.15.66:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f475804f230 con 0x7f476013a430 2024-03-25T18:03:54.788 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.787+0000 7f4756ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:03:54.788 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:54.787+0000 7f4756ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:03:55.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:55.149+0000 7f4764bcd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:03:55.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:55.149+0000 7f4764bcd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:03:55.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:55.149+0000 7f4764bcd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:03:55.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:55.149+0000 7f4764bcd640 1 -- 172.21.15.66:0/811863265 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4760139640 con 0x7f476013a430 2024-03-25T18:03:55.181 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:55.181+0000 7f4756ffd640 1 -- 172.21.15.66:0/811863265 <== mon.2 v1:172.21.15.66:6790/0 10 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f475804ece0 con 0x7f476013a430 2024-03-25T18:03:55.412 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:55.412+0000 7f4756ffd640 1 -- 172.21.15.66:0/811863265 <== mon.2 v1:172.21.15.66:6790/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f4758003970 con 0x7f476013a430 2024-03-25T18:03:55.412 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:55.412+0000 7f4756ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:03:55.412 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:55.412+0000 7f4756ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:03:55.413 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:03:55.413 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:03:55.416 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:55.415+0000 7f4764bcd640 10 librados: watch_flush enter 2024-03-25T18:03:55.416 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:55.415+0000 7f4764bcd640 10 librados: watch_flush exit 2024-03-25T18:03:55.416 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:55.415+0000 7f4764bcd640 20 client.4175.objecter shutdown clearing up homeless session... 2024-03-25T18:03:55.416 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:55.415+0000 7f4764bcd640 10 client.4175.objecter successfully canceled tick 2024-03-25T18:03:55.416 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:55.415+0000 7f4764bcd640 1 -- 172.21.15.66:0/811863265 >> v1:172.21.15.193:6800/2275676803 conn(0x7f474406dd60 legacy=0x7f4744070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:55.416 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:55.415+0000 7f4764bcd640 10 monclient: shutdown 2024-03-25T18:03:55.416 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:55.415+0000 7f4764bcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:03:55.416 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:55.415+0000 7f4764bcd640 1 -- 172.21.15.66:0/811863265 >> v1:172.21.15.66:6790/0 conn(0x7f476013a430 legacy=0x7f47601db360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:03:55.416 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:55.415+0000 7f4764bcd640 1 -- 172.21.15.66:0/811863265 shutdown_connections 2024-03-25T18:03:55.416 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:55.415+0000 7f4764bcd640 1 -- 172.21.15.66:0/811863265 >> 172.21.15.66:0/811863265 conn(0x7f47600b7ea0 msgr2=0x7f4760074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:03:55.416 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:55.415+0000 7f4764bcd640 1 -- 172.21.15.66:0/811863265 shutdown_connections 2024-03-25T18:03:55.416 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:55.415+0000 7f4764bcd640 1 -- 172.21.15.66:0/811863265 wait complete. 2024-03-25T18:03:55.417 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:03:55.415+0000 7f4764bcd640 1 librados: shutdown 2024-03-25T18:03:55.431 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:03:59.343 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:06.733 35227== Warning: unimplemented fcntl command: 1038 2024-03-25T18:03:59.367 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:06.752 35229== Warning: unimplemented fcntl command: 1038 2024-03-25T18:03:59.418 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:06.808 35227== Warning: unimplemented fcntl command: 1038 2024-03-25T18:03:59.480 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:06.799 35230== Warning: unimplemented fcntl command: 1038 2024-03-25T18:03:59.480 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:06.814 35229== Warning: unimplemented fcntl command: 1038 2024-03-25T18:03:59.505 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:06.885 35230== Warning: unimplemented fcntl command: 1038 2024-03-25T18:03:59.682 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:07.068 35228== Warning: unimplemented fcntl command: 1038 2024-03-25T18:03:59.741 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:07.128 35228== Warning: unimplemented fcntl command: 1038 2024-03-25T18:03:59.825 INFO:tasks.ceph.osd.0.smithi066.stderr:==00:00:00:07.196 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:03:59.900 INFO:tasks.ceph.osd.0.smithi066.stderr:==00:00:00:07.271 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:03:59.953 INFO:tasks.ceph.osd.4.smithi193.stderr:2024-03-25T18:03:59.946+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:03:59.981 INFO:tasks.ceph.osd.2.smithi066.stderr:==00:00:00:07.350 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:00.004 INFO:tasks.ceph.osd.6.smithi193.stderr:2024-03-25T18:03:59.998+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:04:00.042 INFO:tasks.ceph.osd.2.smithi066.stderr:==00:00:00:07.411 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:00.056 INFO:tasks.ceph.osd.3.smithi066.stderr:==00:00:00:07.418 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:00.058 INFO:tasks.ceph.osd.7.smithi193.stderr:2024-03-25T18:04:00.051+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:04:00.117 INFO:tasks.ceph.osd.3.smithi066.stderr:==00:00:00:07.479 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:00.168 INFO:tasks.ceph.osd.1.smithi066.stderr:==00:00:00:07.543 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:00.236 INFO:tasks.ceph.osd.1.smithi066.stderr:==00:00:00:07.611 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:00.316 INFO:tasks.ceph.osd.5.smithi193.stderr:2024-03-25T18:04:00.307+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:04:00.434 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:07.824 35227== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:00.447 INFO:tasks.ceph.osd.0.smithi066.stderr:2024-03-25T18:04:00.441+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:04:00.467 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:07.853 35229== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:00.532 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:07.912 35230== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:00.621 INFO:tasks.ceph.osd.2.smithi066.stderr:2024-03-25T18:04:00.614+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:04:00.674 INFO:tasks.ceph.osd.3.smithi066.stderr:2024-03-25T18:04:00.669+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:04:00.776 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:08.162 35228== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:00.803 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:08.193 35227== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:00.829 INFO:tasks.ceph.osd.1.smithi066.stderr:2024-03-25T18:04:00.823+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:04:00.834 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:08.220 35229== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:00.900 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:08.280 35230== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:00.903 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:08.289 35229== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:00.944 INFO:tasks.ceph.osd.0.smithi066.stderr:==00:00:00:08.315 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:01.085 INFO:tasks.ceph.osd.2.smithi066.stderr:==00:00:00:08.454 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:01.141 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:08.527 35228== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:01.167 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:08.553 35229== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:01.175 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:08.565 35227== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:01.178 INFO:tasks.ceph.osd.3.smithi066.stderr:==00:00:00:08.540 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:01.215 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:08.595 35230== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:01.314 INFO:tasks.ceph.osd.0.smithi066.stderr:==00:00:00:08.685 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:01.317 INFO:tasks.ceph.osd.1.smithi066.stderr:==00:00:00:08.693 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:01.436 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:08.822 35229== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:01.460 INFO:tasks.ceph.osd.2.smithi066.stderr:==00:00:00:08.829 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:01.467 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:08.857 35227== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:01.496 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:08.876 35230== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:01.507 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:08.892 35228== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:01.555 INFO:tasks.ceph.osd.3.smithi066.stderr:==00:00:00:08.917 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:01.633 INFO:tasks.ceph.osd.0.smithi066.stderr:==00:00:00:09.004 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:01.685 INFO:tasks.ceph.osd.1.smithi066.stderr:==00:00:00:09.060 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:01.704 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:09.090 35229== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:01.753 INFO:tasks.ceph.osd.1.smithi066.stderr:==00:00:00:09.128 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:01.758 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:09.148 35227== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:01.764 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:09.144 35230== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:01.800 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:09.186 35228== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:01.829 INFO:tasks.ceph.osd.2.smithi066.stderr:==00:00:00:09.197 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:01.873 INFO:tasks.ceph.osd.3.smithi066.stderr:==00:00:00:09.235 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:01.897 INFO:tasks.ceph.osd.0.smithi066.stderr:==00:00:00:09.269 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:01.969 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:09.354 35229== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:02.017 INFO:tasks.ceph.osd.1.smithi066.stderr:==00:00:00:09.393 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:02.028 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:09.408 35230== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:02.052 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:09.442 35227== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:02.090 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:09.476 35228== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:02.118 INFO:tasks.ceph.osd.2.smithi066.stderr:==00:00:00:09.487 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:02.139 INFO:tasks.ceph.osd.3.smithi066.stderr:==00:00:00:09.501 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:02.163 INFO:tasks.ceph.osd.0.smithi066.stderr:==00:00:00:09.535 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:02.232 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:09.618 35229== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:02.237 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:04:02.292 INFO:tasks.ceph.osd.1.smithi066.stderr:==00:00:00:09.668 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:02.294 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:09.674 35230== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:02.329 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.328+0000 7f15cca0f640 10 monclient: get_monmap_and_config 2024-03-25T18:04:02.329 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.328+0000 7f15cca0f640 10 monclient: build_initial_monmap 2024-03-25T18:04:02.329 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.329+0000 7f15cca0f640 10 monclient: monmap: 2024-03-25T18:04:02.329 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:04:02.329 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:02.329 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:04:02.329478+0000 2024-03-25T18:04:02.329 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:04:02.329478+0000 2024-03-25T18:04:02.330 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:02.330 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:02.330 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:04:02.330 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:04:02.330 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:04:02.330 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:02.332 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.329+0000 7f15cca0f640 1 Processor -- start 2024-03-25T18:04:02.332 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.329+0000 7f15cca0f640 1 -- start start 2024-03-25T18:04:02.332 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.329+0000 7f15cca0f640 10 monclient: init 2024-03-25T18:04:02.332 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.330+0000 7f15cca0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:02.332 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.330+0000 7f15cca0f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:04:02.332 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.330+0000 7f15cca0f640 10 monclient(hunting): picked mon.noname-a con 0x7f15c80a96e0 addr v1:172.21.15.66:6789/0 2024-03-25T18:04:02.332 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.330+0000 7f15cca0f640 10 monclient(hunting): picked mon.noname-b con 0x7f15c80b1b10 addr v1:172.21.15.66:6790/0 2024-03-25T18:04:02.332 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.330+0000 7f15cca0f640 10 monclient(hunting): picked mon.noname-c con 0x7f15c8146060 addr v1:172.21.15.193:6789/0 2024-03-25T18:04:02.332 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.330+0000 7f15cca0f640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c8140040 con 0x7f15c80a96e0 2024-03-25T18:04:02.333 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.330+0000 7f15cca0f640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c80b1fa0 con 0x7f15c8146060 2024-03-25T18:04:02.333 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.330+0000 7f15cca0f640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c80b24c0 con 0x7f15c80b1b10 2024-03-25T18:04:02.333 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.330+0000 7f15cca0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:02.333 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.330+0000 7f15cca0f640 10 monclient(hunting): authenticate will time out at 6991.389573s 2024-03-25T18:04:02.333 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.331+0000 7f15c5d9b640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f15c80b1b10 0x7f15c8145950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:52590/0 (socket says 172.21.15.66:52590) 2024-03-25T18:04:02.333 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.331+0000 7f15c5d9b640 1 -- 172.21.15.66:0/2749295649 learned_addr learned my addr 172.21.15.66:0/2749295649 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:04:02.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.333+0000 7f15c559a640 1 -- 172.21.15.66:0/2749295649 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 637177201 0 0) 0x7f15c80b1fa0 con 0x7f15c8146060 2024-03-25T18:04:02.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.333+0000 7f15c559a640 10 monclient(hunting): _negotiate 2024-03-25T18:04:02.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.333+0000 7f15c559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:02.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.333+0000 7f15c559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:02.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.333+0000 7f15c559a640 10 monclient(hunting): my global_id is 4201 2024-03-25T18:04:02.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.333+0000 7f15c559a640 1 -- 172.21.15.66:0/2749295649 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15b0003520 con 0x7f15c8146060 2024-03-25T18:04:02.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.333+0000 7f15c559a640 1 -- 172.21.15.66:0/2749295649 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 680085709 0 0) 0x7f15c8140040 con 0x7f15c80a96e0 2024-03-25T18:04:02.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.333+0000 7f15c559a640 10 monclient(hunting): _negotiate 2024-03-25T18:04:02.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.333+0000 7f15c559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:02.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.333+0000 7f15c559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:02.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.333+0000 7f15c559a640 10 monclient(hunting): my global_id is 4203 2024-03-25T18:04:02.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.333+0000 7f15c559a640 1 -- 172.21.15.66:0/2749295649 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15c80b1fa0 con 0x7f15c80a96e0 2024-03-25T18:04:02.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.333+0000 7f15c559a640 1 -- 172.21.15.66:0/2749295649 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2850702288 0 0) 0x7f15c80b24c0 con 0x7f15c80b1b10 2024-03-25T18:04:02.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.333+0000 7f15c559a640 10 monclient(hunting): _negotiate 2024-03-25T18:04:02.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.333+0000 7f15c559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:02.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.333+0000 7f15c559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:02.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.333+0000 7f15c559a640 10 monclient(hunting): my global_id is 4202 2024-03-25T18:04:02.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.333+0000 7f15c559a640 1 -- 172.21.15.66:0/2749295649 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15c8140040 con 0x7f15c80b1b10 2024-03-25T18:04:02.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.336+0000 7f15c559a640 1 -- 172.21.15.66:0/2749295649 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1982263654 0 0) 0x7f15b0003520 con 0x7f15c8146060 2024-03-25T18:04:02.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.336+0000 7f15c559a640 1 -- 172.21.15.66:0/2749295649 >> v1:172.21.15.66:6790/0 conn(0x7f15c80b1b10 legacy=0x7f15c8145950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:02.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.336+0000 7f15c559a640 1 -- 172.21.15.66:0/2749295649 >> v1:172.21.15.66:6789/0 conn(0x7f15c80a96e0 legacy=0x7f15c80b0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:02.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.336+0000 7f15c559a640 10 monclient: _finish_hunting 0 2024-03-25T18:04:02.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.336+0000 7f15c559a640 1 monclient: found mon.noname-c 2024-03-25T18:04:02.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.336+0000 7f15c559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:02.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.336+0000 7f15c559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:04:02.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.336+0000 7f15c559a640 1 -- 172.21.15.66:0/2749295649 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c80b15b0 con 0x7f15c8146060 2024-03-25T18:04:02.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.336+0000 7f15c559a640 10 monclient: _finish_auth 0 2024-03-25T18:04:02.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.336+0000 7f15c559a640 10 monclient: _check_auth_tickets 2024-03-25T18:04:02.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.336+0000 7f15c559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:02.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.336+0000 7f15cca0f640 5 monclient: authenticate success, global_id 4201 2024-03-25T18:04:02.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.336+0000 7f15cca0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:04:02.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.336+0000 7f15c559a640 1 -- 172.21.15.66:0/2749295649 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f15c0001900 con 0x7f15c8146060 2024-03-25T18:04:02.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.336+0000 7f15c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:02.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.336+0000 7f15c559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:04:02.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.336+0000 7f15c559a640 10 monclient: dump: 2024-03-25T18:04:02.339 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:02.340 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:02.340 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:02.340 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:02.340 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:02.340 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:02.340 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:02.340 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:02.340 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:02.340 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:02.340 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.336+0000 7f15cca0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:04:02.345 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.338+0000 7f15c559a640 1 -- 172.21.15.66:0/2749295649 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15c0001b50 con 0x7f15c8146060 2024-03-25T18:04:02.345 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.338+0000 7f15c559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:02.345 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.338+0000 7f15c559a640 1 -- 172.21.15.66:0/2749295649 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f15c0001ee0 con 0x7f15c8146060 2024-03-25T18:04:02.345 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.338+0000 7f15cca0f640 10 monclient: get_monmap_and_config success 2024-03-25T18:04:02.345 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.338+0000 7f15cca0f640 4 set_mon_vals no callback set 2024-03-25T18:04:02.345 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.338+0000 7f15c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:02.345 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.338+0000 7f15c559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:04:02.346 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.338+0000 7f15c559a640 10 monclient: dump: 2024-03-25T18:04:02.346 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:02.346 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:02.346 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:02.346 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:02.346 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:02.346 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:02.346 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:02.346 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:02.347 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:02.347 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:02.347 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.339+0000 7f15cca0f640 10 monclient: shutdown 2024-03-25T18:04:02.347 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.339+0000 7f15cca0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:02.347 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.339+0000 7f15cca0f640 1 -- 172.21.15.66:0/2749295649 >> v1:172.21.15.193:6789/0 conn(0x7f15c8146060 legacy=0x7f15c814dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:02.347 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.339+0000 7f15cca0f640 1 -- 172.21.15.66:0/2749295649 shutdown_connections 2024-03-25T18:04:02.347 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.339+0000 7f15cca0f640 1 -- 172.21.15.66:0/2749295649 >> 172.21.15.66:0/2749295649 conn(0x7f15c80b7ea0 msgr2=0x7f15c80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:02.347 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.339+0000 7f15cca0f640 1 -- 172.21.15.66:0/2749295649 shutdown_connections 2024-03-25T18:04:02.348 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.339+0000 7f15cca0f640 1 -- 172.21.15.66:0/2749295649 wait complete. 2024-03-25T18:04:02.348 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.340+0000 7f15cca0f640 10 monclient: build_initial_monmap 2024-03-25T18:04:02.348 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.340+0000 7f15cca0f640 10 monclient: monmap: 2024-03-25T18:04:02.348 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:04:02.348 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:04:02.348 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:04:02.348 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:04:02.348 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:02.348 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:04:02.348 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:04:02.349 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:04:02.349 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:04:02.349 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:02.349 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.340+0000 7f15cca0f640 1 librados: starting msgr at 2024-03-25T18:04:02.349 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.340+0000 7f15cca0f640 1 librados: starting objecter 2024-03-25T18:04:02.349 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.340+0000 7f15cca0f640 1 Processor -- start 2024-03-25T18:04:02.349 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.340+0000 7f15cca0f640 1 -- start start 2024-03-25T18:04:02.349 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.340+0000 7f15cca0f640 1 librados: setting wanted keys 2024-03-25T18:04:02.349 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.340+0000 7f15cca0f640 1 librados: calling monclient init 2024-03-25T18:04:02.349 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.340+0000 7f15cca0f640 10 monclient: init 2024-03-25T18:04:02.349 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.341+0000 7f15cca0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:02.350 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.341+0000 7f15cca0f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:04:02.350 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.341+0000 7f15cca0f640 10 monclient(hunting): picked mon.noname-a con 0x7f15c80a96e0 addr v1:172.21.15.66:6789/0 2024-03-25T18:04:02.350 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.341+0000 7f15cca0f640 10 monclient(hunting): picked mon.noname-b con 0x7f15c8146060 addr v1:172.21.15.193:6789/0 2024-03-25T18:04:02.350 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.341+0000 7f15cca0f640 10 monclient(hunting): picked mon.noname-c con 0x7f15c81cc210 addr v1:172.21.15.66:6790/0 2024-03-25T18:04:02.350 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.341+0000 7f15cca0f640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c81ce670 con 0x7f15c80a96e0 2024-03-25T18:04:02.350 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.341+0000 7f15cca0f640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c81cf1e0 con 0x7f15c8146060 2024-03-25T18:04:02.350 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.341+0000 7f15cca0f640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15c81d89d0 con 0x7f15c81cc210 2024-03-25T18:04:02.350 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.341+0000 7f15cca0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:02.350 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.341+0000 7f15cca0f640 10 monclient(hunting): authenticate will time out at 6991.400414s 2024-03-25T18:04:02.351 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.341+0000 7f15c6d9d640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f15c81cc210 0x7f15c81e2d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:52596/0 (socket says 172.21.15.66:52596) 2024-03-25T18:04:02.351 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.341+0000 7f15c659c640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f15c80a96e0 0x7f15c8139f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:46816/0 (socket says 172.21.15.66:46816) 2024-03-25T18:04:02.351 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.341+0000 7f15c6d9d640 1 -- 172.21.15.66:0/345395564 learned_addr learned my addr 172.21.15.66:0/345395564 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:04:02.351 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.342+0000 7f15b6ffd640 10 client.?.objecter ms_handle_connect 0x7f15c81cc210 2024-03-25T18:04:02.351 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.342+0000 7f15b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:02.351 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.342+0000 7f15b6ffd640 10 client.?.objecter ms_handle_connect 0x7f15c80a96e0 2024-03-25T18:04:02.351 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.342+0000 7f15b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:02.351 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.342+0000 7f15b6ffd640 10 client.?.objecter ms_handle_connect 0x7f15c8146060 2024-03-25T18:04:02.351 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.342+0000 7f15b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:02.351 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.343+0000 7f15b6ffd640 1 -- 172.21.15.66:0/345395564 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 203056796 0 0) 0x7f15c81cf1e0 con 0x7f15c8146060 2024-03-25T18:04:02.351 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.343+0000 7f15b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:04:02.351 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.343+0000 7f15b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:02.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.343+0000 7f15b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:02.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.343+0000 7f15b6ffd640 10 monclient(hunting): my global_id is 4204 2024-03-25T18:04:02.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.343+0000 7f15b6ffd640 1 -- 172.21.15.66:0/345395564 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15a8004eb0 con 0x7f15c8146060 2024-03-25T18:04:02.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.344+0000 7f15b6ffd640 1 -- 172.21.15.66:0/345395564 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4272543288 0 0) 0x7f15c81ce670 con 0x7f15c80a96e0 2024-03-25T18:04:02.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.344+0000 7f15b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:04:02.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.344+0000 7f15b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:02.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.344+0000 7f15b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:02.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.344+0000 7f15b6ffd640 10 monclient(hunting): my global_id is 4206 2024-03-25T18:04:02.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.344+0000 7f15b6ffd640 1 -- 172.21.15.66:0/345395564 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15c81cf1e0 con 0x7f15c80a96e0 2024-03-25T18:04:02.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.344+0000 7f15b6ffd640 1 -- 172.21.15.66:0/345395564 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2631483160 0 0) 0x7f15c81d89d0 con 0x7f15c81cc210 2024-03-25T18:04:02.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.344+0000 7f15b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:04:02.353 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.344+0000 7f15b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:02.353 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.344+0000 7f15b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:02.353 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.344+0000 7f15b6ffd640 10 monclient(hunting): my global_id is 4205 2024-03-25T18:04:02.353 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.344+0000 7f15b6ffd640 1 -- 172.21.15.66:0/345395564 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15c81ce670 con 0x7f15c81cc210 2024-03-25T18:04:02.354 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:09.733 35227== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:02.356 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.348+0000 7f15b6ffd640 1 -- 172.21.15.66:0/345395564 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1819371523 0 0) 0x7f15a8004eb0 con 0x7f15c8146060 2024-03-25T18:04:02.356 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.348+0000 7f15b6ffd640 1 -- 172.21.15.66:0/345395564 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15c81d89d0 con 0x7f15c8146060 2024-03-25T18:04:02.356 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.348+0000 7f15b6ffd640 1 -- 172.21.15.66:0/345395564 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f15bc001a70 con 0x7f15c8146060 2024-03-25T18:04:02.356 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.348+0000 7f15b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:02.356 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.348+0000 7f15b6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:04:02.356 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.348+0000 7f15b6ffd640 10 monclient(hunting): dump: 2024-03-25T18:04:02.357 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:02.357 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:02.357 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:02.357 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:02.357 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:02.357 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:02.357 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:02.357 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:02.357 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:02.357 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:02.357 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.348+0000 7f15b6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:04:02.358 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.349+0000 7f15b6ffd640 1 -- 172.21.15.66:0/345395564 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 327883289 0 0) 0x7f15c81ce670 con 0x7f15c81cc210 2024-03-25T18:04:02.358 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.349+0000 7f15b6ffd640 1 -- 172.21.15.66:0/345395564 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15a8004eb0 con 0x7f15c81cc210 2024-03-25T18:04:02.358 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.349+0000 7f15b6ffd640 1 -- 172.21.15.66:0/345395564 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f15c0001e60 con 0x7f15c81cc210 2024-03-25T18:04:02.358 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.349+0000 7f15b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:02.358 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.349+0000 7f15b6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:04:02.358 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.349+0000 7f15b6ffd640 10 monclient(hunting): dump: 2024-03-25T18:04:02.358 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:02.358 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:02.358 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:02.358 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:02.358 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:02.359 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:02.359 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:02.359 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:02.359 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:02.359 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:02.359 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.350+0000 7f15b6ffd640 1 -- 172.21.15.66:0/345395564 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3462599608 0 0) 0x7f15c81cf1e0 con 0x7f15c80a96e0 2024-03-25T18:04:02.359 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.350+0000 7f15b6ffd640 1 -- 172.21.15.66:0/345395564 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15c81ce670 con 0x7f15c80a96e0 2024-03-25T18:04:02.359 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.350+0000 7f15b6ffd640 1 -- 172.21.15.66:0/345395564 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f15b80018f0 con 0x7f15c80a96e0 2024-03-25T18:04:02.359 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.350+0000 7f15b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:02.359 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.350+0000 7f15b6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:04:02.359 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.350+0000 7f15b6ffd640 10 monclient(hunting): dump: 2024-03-25T18:04:02.360 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:02.360 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:02.360 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:02.360 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:02.360 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:02.360 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:02.360 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:02.360 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:02.360 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:02.360 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:02.360 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.353+0000 7f15b6ffd640 1 -- 172.21.15.66:0/345395564 <== mon.2 v1:172.21.15.66:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2101048815 0 0) 0x7f15a8004eb0 con 0x7f15c81cc210 2024-03-25T18:04:02.361 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.353+0000 7f15b6ffd640 1 -- 172.21.15.66:0/345395564 >> v1:172.21.15.193:6789/0 conn(0x7f15c8146060 legacy=0x7f15c8142830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:02.361 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.353+0000 7f15b6ffd640 1 -- 172.21.15.66:0/345395564 >> v1:172.21.15.66:6789/0 conn(0x7f15c80a96e0 legacy=0x7f15c8139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:02.361 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.353+0000 7f15b6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:04:02.361 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.353+0000 7f15b6ffd640 1 monclient: found mon.c 2024-03-25T18:04:02.361 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.353+0000 7f15b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:02.361 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.353+0000 7f15b6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:04:02.361 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.353+0000 7f15b6ffd640 1 -- 172.21.15.66:0/345395564 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c81d8b60 con 0x7f15c81cc210 2024-03-25T18:04:02.361 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.353+0000 7f15b6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:04:02.361 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.353+0000 7f15b6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:04:02.361 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.353+0000 7f15b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:02.361 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.353+0000 7f15cca0f640 5 monclient: authenticate success, global_id 4205 2024-03-25T18:04:02.361 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.353+0000 7f15cca0f640 10 monclient: _renew_subs 2024-03-25T18:04:02.361 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.353+0000 7f15cca0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:04:02.362 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.353+0000 7f15cca0f640 1 -- 172.21.15.66:0/345395564 --> v1:172.21.15.66:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15c813acc0 con 0x7f15c81cc210 2024-03-25T18:04:02.362 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.353+0000 7f15cca0f640 10 client.4205.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:04:02.362 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.353+0000 7f15cca0f640 10 monclient: _renew_subs 2024-03-25T18:04:02.362 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.353+0000 7f15cca0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:04:02.362 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.353+0000 7f15cca0f640 1 -- 172.21.15.66:0/345395564 --> v1:172.21.15.66:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15c813b310 con 0x7f15c81cc210 2024-03-25T18:04:02.362 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.353+0000 7f15cca0f640 1 librados: init done 2024-03-25T18:04:02.362 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.354+0000 7f15cca0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:02.362 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.354+0000 7f15cca0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:02.362 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.354+0000 7f15cca0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:04:02.362 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.354+0000 7f15cca0f640 1 -- 172.21.15.66:0/345395564 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15c80a9ac0 con 0x7f15c81cc210 2024-03-25T18:04:02.382 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:09.768 35228== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:02.415 INFO:tasks.ceph.osd.3.smithi066.stderr:==00:00:00:09.777 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:02.420 INFO:tasks.ceph.osd.2.smithi066.stderr:==00:00:00:09.789 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:02.437 INFO:tasks.ceph.osd.0.smithi066.stderr:==00:00:00:09.808 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:02.496 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:09.882 35229== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:02.558 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:09.938 35230== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:02.560 INFO:tasks.ceph.osd.1.smithi066.stderr:==00:00:00:09.936 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:02.615 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.615+0000 7f15b6ffd640 1 -- 172.21.15.66:0/345395564 <== mon.2 v1:172.21.15.66:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15c00020c0 con 0x7f15c81cc210 2024-03-25T18:04:02.615 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.615+0000 7f15b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:02.615 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.615+0000 7f15b6ffd640 1 -- 172.21.15.66:0/345395564 <== mon.2 v1:172.21.15.66:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f15c0003c20 con 0x7f15c81cc210 2024-03-25T18:04:02.615 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.615+0000 7f15b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:02.616 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.615+0000 7f15b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:04:02.616 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.615+0000 7f15b6ffd640 10 monclient: dump: 2024-03-25T18:04:02.616 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:02.616 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:02.616 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:02.616 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:02.616 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:02.616 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:02.616 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:02.616 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:02.616 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:02.617 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:02.617 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.615+0000 7f15c4d99640 4 set_mon_vals no callback set 2024-03-25T18:04:02.617 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.615+0000 7f15b6ffd640 1 -- 172.21.15.66:0/345395564 <== mon.2 v1:172.21.15.66:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f15c0003e40 con 0x7f15c81cc210 2024-03-25T18:04:02.618 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.616+0000 7f15c659c640 10 monclient: get_auth_request con 0x7f15a806e180 auth_method 0 2024-03-25T18:04:02.618 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.616+0000 7f15b6ffd640 10 client.4205.objecter ms_handle_connect 0x7f15a806e180 2024-03-25T18:04:02.622 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.622+0000 7f15b6ffd640 1 -- 172.21.15.66:0/345395564 <== mon.2 v1:172.21.15.66:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f15c0001490 con 0x7f15c81cc210 2024-03-25T18:04:02.622 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.622+0000 7f15b6ffd640 10 client.4205.objecter ms_dispatch 0x7f15c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:04:02.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.622+0000 7f15b6ffd640 3 client.4205.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:04:02.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.622+0000 7f15b6ffd640 3 client.4205.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:04:02.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.622+0000 7f15b6ffd640 20 client.4205.objecter dump_active .. 0 homeless 2024-03-25T18:04:02.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.622+0000 7f15b6ffd640 1 -- 172.21.15.66:0/345395564 <== mon.2 v1:172.21.15.66:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f15c00503e0 con 0x7f15c81cc210 2024-03-25T18:04:02.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.622+0000 7f15b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:02.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.622+0000 7f15b6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:04:02.637 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:10.027 35227== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:02.670 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:10.056 35228== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:02.677 INFO:tasks.ceph.osd.3.smithi066.stderr:==00:00:00:10.039 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:02.710 INFO:tasks.ceph.osd.2.smithi066.stderr:==00:00:00:10.079 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:02.711 INFO:tasks.ceph.osd.0.smithi066.stderr:==00:00:00:10.082 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:02.803 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.803+0000 7f15cca0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:02.804 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.803+0000 7f15cca0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:02.804 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.803+0000 7f15cca0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:04:02.804 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.803+0000 7f15cca0f640 1 -- 172.21.15.66:0/345395564 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15c80627b0 con 0x7f15c81cc210 2024-03-25T18:04:02.819 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.818+0000 7f15b6ffd640 1 -- 172.21.15.66:0/345395564 <== mon.2 v1:172.21.15.66:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f15c0052310 con 0x7f15c81cc210 2024-03-25T18:04:02.819 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.818+0000 7f15b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:02.819 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.818+0000 7f15b6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:04:02.819 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:04:02.819 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:04:02.822 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.821+0000 7f15cca0f640 10 librados: watch_flush enter 2024-03-25T18:04:02.822 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.821+0000 7f15cca0f640 10 librados: watch_flush exit 2024-03-25T18:04:02.822 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.821+0000 7f15cca0f640 20 client.4205.objecter shutdown clearing up homeless session... 2024-03-25T18:04:02.822 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.821+0000 7f15cca0f640 10 client.4205.objecter successfully canceled tick 2024-03-25T18:04:02.822 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.821+0000 7f15cca0f640 1 -- 172.21.15.66:0/345395564 >> v1:172.21.15.193:6800/2275676803 conn(0x7f15a806e180 legacy=0x7f15a8070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:02.822 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.821+0000 7f15cca0f640 10 monclient: shutdown 2024-03-25T18:04:02.823 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.821+0000 7f15cca0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:02.823 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.821+0000 7f15cca0f640 1 -- 172.21.15.66:0/345395564 >> v1:172.21.15.66:6790/0 conn(0x7f15c81cc210 legacy=0x7f15c81e2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:02.823 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.821+0000 7f15cca0f640 1 -- 172.21.15.66:0/345395564 shutdown_connections 2024-03-25T18:04:02.823 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.821+0000 7f15cca0f640 1 -- 172.21.15.66:0/345395564 >> 172.21.15.66:0/345395564 conn(0x7f15c80b7ea0 msgr2=0x7f15c813dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:02.823 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.821+0000 7f15cca0f640 1 -- 172.21.15.66:0/345395564 shutdown_connections 2024-03-25T18:04:02.823 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.821+0000 7f15cca0f640 1 -- 172.21.15.66:0/345395564 wait complete. 2024-03-25T18:04:02.823 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:02.821+0000 7f15cca0f640 1 librados: shutdown 2024-03-25T18:04:02.830 INFO:tasks.ceph.osd.1.smithi066.stderr:==00:00:00:10.206 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:02.832 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:04:02.832 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:10.212 35230== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:02.924 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:10.314 35227== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:02.946 INFO:tasks.ceph.osd.3.smithi066.stderr:==00:00:00:10.308 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:02.962 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:10.348 35228== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:02.982 INFO:tasks.ceph.osd.0.smithi066.stderr:==00:00:00:10.353 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:02.995 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:10.381 35229== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:03.006 INFO:tasks.ceph.osd.2.smithi066.stderr:==00:00:00:10.375 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:03.047 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:10.433 35229== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:03.099 INFO:tasks.ceph.osd.1.smithi066.stderr:==00:00:00:10.474 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:03.211 INFO:tasks.ceph.osd.3.smithi066.stderr:==00:00:00:10.573 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:03.248 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:10.635 35228== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:03.255 INFO:tasks.ceph.osd.0.smithi066.stderr:==00:00:00:10.627 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:03.295 INFO:tasks.ceph.osd.2.smithi066.stderr:==00:00:00:10.664 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:03.301 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:10.681 35230== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:03.356 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:10.736 35230== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:03.369 INFO:tasks.ceph.osd.1.smithi066.stderr:==00:00:00:10.745 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:03.452 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:10.842 35227== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:03.475 INFO:tasks.ceph.osd.3.smithi066.stderr:==00:00:00:10.837 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:03.518 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:10.908 35227== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:03.590 INFO:tasks.ceph.osd.2.smithi066.stderr:==00:00:00:10.959 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:03.746 INFO:tasks.ceph.osd.0.smithi066.stderr:==00:00:00:11.118 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:03.807 INFO:tasks.ceph.osd.0.smithi066.stderr:==00:00:00:11.178 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:03.808 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:11.194 35228== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:03.863 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:11.248 35228== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:03.897 INFO:tasks.ceph.osd.1.smithi066.stderr:==00:00:00:11.273 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:03.954 INFO:tasks.ceph.osd.1.smithi066.stderr:==00:00:00:11.329 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:03.973 INFO:tasks.ceph.osd.3.smithi066.stderr:==00:00:00:11.335 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:04.041 INFO:tasks.ceph.osd.3.smithi066.stderr:==00:00:00:11.403 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:04.188 INFO:tasks.ceph.osd.2.smithi066.stderr:==00:00:00:11.557 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:04.257 INFO:tasks.ceph.osd.2.smithi066.stderr:==00:00:00:11.626 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:08.503 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:15.888 35229== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:09.181 INFO:tasks.ceph.osd.0.smithi066.stderr:==00:00:00:16.553 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:09.344 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:16.724 35230== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:09.354 INFO:tasks.ceph.osd.3.smithi066.stderr:==00:00:00:16.716 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:09.375 INFO:tasks.ceph.osd.1.smithi066.stderr:==00:00:00:16.751 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:09.399 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:16.789 35227== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:09.638 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:04:09.729 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.729+0000 7f1770f10640 10 monclient: get_monmap_and_config 2024-03-25T18:04:09.730 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.729+0000 7f1770f10640 10 monclient: build_initial_monmap 2024-03-25T18:04:09.730 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.729+0000 7f1770f10640 10 monclient: monmap: 2024-03-25T18:04:09.730 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:04:09.730 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:09.730 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:04:09.730207+0000 2024-03-25T18:04:09.730 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:04:09.730207+0000 2024-03-25T18:04:09.730 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:09.730 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:09.730 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:04:09.730 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:04:09.730 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:04:09.731 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:09.732 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.730+0000 7f1770f10640 1 Processor -- start 2024-03-25T18:04:09.732 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.730+0000 7f1770f10640 1 -- start start 2024-03-25T18:04:09.732 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.730+0000 7f1770f10640 10 monclient: init 2024-03-25T18:04:09.732 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.731+0000 7f1770f10640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:09.733 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.731+0000 7f1770f10640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:04:09.733 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.731+0000 7f1770f10640 10 monclient(hunting): picked mon.noname-a con 0x7f176c0b4d60 addr v1:172.21.15.66:6789/0 2024-03-25T18:04:09.733 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.731+0000 7f1770f10640 10 monclient(hunting): picked mon.noname-b con 0x7f176c0b1730 addr v1:172.21.15.66:6790/0 2024-03-25T18:04:09.733 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.731+0000 7f1770f10640 10 monclient(hunting): picked mon.noname-c con 0x7f176c14be90 addr v1:172.21.15.193:6789/0 2024-03-25T18:04:09.733 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.731+0000 7f1770f10640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f176c0af940 con 0x7f176c0b4d60 2024-03-25T18:04:09.733 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.731+0000 7f1770f10640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f176c0b0a70 con 0x7f176c14be90 2024-03-25T18:04:09.733 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.731+0000 7f1770f10640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f176c0b0d50 con 0x7f176c0b1730 2024-03-25T18:04:09.733 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.731+0000 7f1770f10640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:09.733 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.731+0000 7f1770f10640 10 monclient(hunting): authenticate will time out at 6998.790460s 2024-03-25T18:04:09.733 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.731+0000 7f1769d9b640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f176c0b1730 0x7f176c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:53872/0 (socket says 172.21.15.66:53872) 2024-03-25T18:04:09.733 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.731+0000 7f1769d9b640 1 -- 172.21.15.66:0/1875342414 learned_addr learned my addr 172.21.15.66:0/1875342414 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:04:09.735 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.735+0000 7f176959a640 1 -- 172.21.15.66:0/1875342414 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34085679 0 0) 0x7f176c0b0a70 con 0x7f176c14be90 2024-03-25T18:04:09.735 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.735+0000 7f176959a640 10 monclient(hunting): _negotiate 2024-03-25T18:04:09.735 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.735+0000 7f176959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:09.735 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.735+0000 7f176959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:09.735 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.735+0000 7f176959a640 10 monclient(hunting): my global_id is 4207 2024-03-25T18:04:09.735 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.735+0000 7f176959a640 1 -- 172.21.15.66:0/1875342414 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1754003520 con 0x7f176c14be90 2024-03-25T18:04:09.735 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.735+0000 7f176959a640 1 -- 172.21.15.66:0/1875342414 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3825677091 0 0) 0x7f176c0af940 con 0x7f176c0b4d60 2024-03-25T18:04:09.736 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.735+0000 7f176959a640 10 monclient(hunting): _negotiate 2024-03-25T18:04:09.736 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.735+0000 7f176959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:09.736 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.735+0000 7f176959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:09.736 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.735+0000 7f176959a640 10 monclient(hunting): my global_id is 4209 2024-03-25T18:04:09.736 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.735+0000 7f176959a640 1 -- 172.21.15.66:0/1875342414 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f176c0b0a70 con 0x7f176c0b4d60 2024-03-25T18:04:09.736 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.736+0000 7f176959a640 1 -- 172.21.15.66:0/1875342414 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4109165717 0 0) 0x7f176c0b0d50 con 0x7f176c0b1730 2024-03-25T18:04:09.736 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.736+0000 7f176959a640 10 monclient(hunting): _negotiate 2024-03-25T18:04:09.737 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.736+0000 7f176959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:09.737 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.736+0000 7f176959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:09.737 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.736+0000 7f176959a640 10 monclient(hunting): my global_id is 4208 2024-03-25T18:04:09.737 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.736+0000 7f176959a640 1 -- 172.21.15.66:0/1875342414 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f176c0af940 con 0x7f176c0b1730 2024-03-25T18:04:09.739 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.739+0000 7f176959a640 1 -- 172.21.15.66:0/1875342414 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2559614874 0 0) 0x7f176c0b0a70 con 0x7f176c0b4d60 2024-03-25T18:04:09.739 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.739+0000 7f176959a640 1 -- 172.21.15.66:0/1875342414 >> v1:172.21.15.66:6790/0 conn(0x7f176c0b1730 legacy=0x7f176c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:09.739 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.739+0000 7f176959a640 1 -- 172.21.15.66:0/1875342414 >> v1:172.21.15.193:6789/0 conn(0x7f176c14be90 legacy=0x7f176c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:09.739 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.739+0000 7f176959a640 10 monclient: _finish_hunting 0 2024-03-25T18:04:09.739 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.739+0000 7f176959a640 1 monclient: found mon.noname-a 2024-03-25T18:04:09.739 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.739+0000 7f176959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:09.739 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.739+0000 7f176959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.66:6789/0 2024-03-25T18:04:09.739 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.739+0000 7f176959a640 1 -- 172.21.15.66:0/1875342414 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f176c0b1030 con 0x7f176c0b4d60 2024-03-25T18:04:09.739 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.739+0000 7f176959a640 10 monclient: _finish_auth 0 2024-03-25T18:04:09.739 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.739+0000 7f176959a640 10 monclient: _check_auth_tickets 2024-03-25T18:04:09.740 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.739+0000 7f176959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:09.740 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.739+0000 7f176959a640 1 -- 172.21.15.66:0/1875342414 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f175c001670 con 0x7f176c0b4d60 2024-03-25T18:04:09.740 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.739+0000 7f176959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:09.740 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.739+0000 7f176959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:04:09.740 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.739+0000 7f176959a640 10 monclient: dump: 2024-03-25T18:04:09.740 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:09.740 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:09.740 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:09.740 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:09.740 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:09.740 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:09.740 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:09.740 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:09.740 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:09.740 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:09.740 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.739+0000 7f1770f10640 5 monclient: authenticate success, global_id 4209 2024-03-25T18:04:09.741 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.739+0000 7f1770f10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:04:09.742 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.741+0000 7f176959a640 1 -- 172.21.15.66:0/1875342414 <== mon.0 v1:172.21.15.66:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f175c0017d0 con 0x7f176c0b4d60 2024-03-25T18:04:09.742 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.741+0000 7f176959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:09.742 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.741+0000 7f176959a640 1 -- 172.21.15.66:0/1875342414 <== mon.0 v1:172.21.15.66:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f175c001be0 con 0x7f176c0b4d60 2024-03-25T18:04:09.742 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.741+0000 7f176959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:09.742 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.741+0000 7f176959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:04:09.742 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.741+0000 7f176959a640 10 monclient: dump: 2024-03-25T18:04:09.742 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:09.742 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:09.742 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:09.742 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:09.742 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:09.742 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:09.742 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:09.743 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:09.743 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:09.743 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:09.743 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.742+0000 7f1770f10640 10 monclient: get_monmap_and_config success 2024-03-25T18:04:09.743 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.742+0000 7f1770f10640 4 set_mon_vals no callback set 2024-03-25T18:04:09.744 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.742+0000 7f1770f10640 10 monclient: shutdown 2024-03-25T18:04:09.744 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.742+0000 7f1770f10640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:09.744 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.742+0000 7f1770f10640 1 -- 172.21.15.66:0/1875342414 >> v1:172.21.15.66:6789/0 conn(0x7f176c0b4d60 legacy=0x7f176c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:09.744 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.743+0000 7f1770f10640 1 -- 172.21.15.66:0/1875342414 shutdown_connections 2024-03-25T18:04:09.744 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.743+0000 7f1770f10640 1 -- 172.21.15.66:0/1875342414 >> 172.21.15.66:0/1875342414 conn(0x7f176c0617a0 msgr2=0x7f176c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:09.744 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.743+0000 7f1770f10640 1 -- 172.21.15.66:0/1875342414 shutdown_connections 2024-03-25T18:04:09.744 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.743+0000 7f1770f10640 1 -- 172.21.15.66:0/1875342414 wait complete. 2024-03-25T18:04:09.745 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.744+0000 7f1770f10640 10 monclient: build_initial_monmap 2024-03-25T18:04:09.745 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.744+0000 7f1770f10640 10 monclient: monmap: 2024-03-25T18:04:09.745 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:04:09.745 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:04:09.745 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:04:09.745 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:04:09.746 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:09.746 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:04:09.746 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:04:09.746 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:04:09.746 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:04:09.746 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:09.746 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.745+0000 7f1770f10640 1 librados: starting msgr at 2024-03-25T18:04:09.746 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.745+0000 7f1770f10640 1 librados: starting objecter 2024-03-25T18:04:09.746 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.745+0000 7f1770f10640 1 Processor -- start 2024-03-25T18:04:09.746 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.745+0000 7f1770f10640 1 -- start start 2024-03-25T18:04:09.746 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.745+0000 7f1770f10640 1 librados: setting wanted keys 2024-03-25T18:04:09.746 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.745+0000 7f1770f10640 1 librados: calling monclient init 2024-03-25T18:04:09.746 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.745+0000 7f1770f10640 10 monclient: init 2024-03-25T18:04:09.748 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.746+0000 7f1770f10640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:09.748 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.746+0000 7f1770f10640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:04:09.748 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.746+0000 7f1770f10640 10 monclient(hunting): picked mon.noname-b con 0x7f176c0b1730 addr v1:172.21.15.193:6789/0 2024-03-25T18:04:09.748 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.746+0000 7f1770f10640 10 monclient(hunting): picked mon.noname-c con 0x7f176c0b4d60 addr v1:172.21.15.66:6790/0 2024-03-25T18:04:09.749 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.746+0000 7f1770f10640 10 monclient(hunting): picked mon.noname-a con 0x7f176c14be90 addr v1:172.21.15.66:6789/0 2024-03-25T18:04:09.749 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.746+0000 7f1770f10640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f176c13a960 con 0x7f176c14be90 2024-03-25T18:04:09.749 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.747+0000 7f1770f10640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f176c1d8540 con 0x7f176c0b1730 2024-03-25T18:04:09.749 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.747+0000 7f1770f10640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f176c142d80 con 0x7f176c0b4d60 2024-03-25T18:04:09.749 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.747+0000 7f1770f10640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:09.749 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.747+0000 7f1770f10640 10 monclient(hunting): authenticate will time out at 6998.806365s 2024-03-25T18:04:09.749 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.747+0000 7f1769d9b640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f176c0b4d60 0x7f176c139290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:53884/0 (socket says 172.21.15.66:53884) 2024-03-25T18:04:09.749 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.747+0000 7f176ad9d640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f176c14be90 0x7f176c13b8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:49794/0 (socket says 172.21.15.66:49794) 2024-03-25T18:04:09.749 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.747+0000 7f1769d9b640 1 -- 172.21.15.66:0/3007396598 learned_addr learned my addr 172.21.15.66:0/3007396598 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:04:09.749 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.747+0000 7f175affd640 10 client.?.objecter ms_handle_connect 0x7f176c0b4d60 2024-03-25T18:04:09.749 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.747+0000 7f175affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:09.749 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.747+0000 7f175affd640 10 client.?.objecter ms_handle_connect 0x7f176c14be90 2024-03-25T18:04:09.749 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.747+0000 7f175affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:09.750 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.748+0000 7f175affd640 10 client.?.objecter ms_handle_connect 0x7f176c0b1730 2024-03-25T18:04:09.750 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.748+0000 7f175affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:09.752 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.750+0000 7f175affd640 1 -- 172.21.15.66:0/3007396598 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3303451359 0 0) 0x7f176c142d80 con 0x7f176c0b4d60 2024-03-25T18:04:09.752 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.750+0000 7f175affd640 10 monclient(hunting): _negotiate 2024-03-25T18:04:09.752 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.750+0000 7f175affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:09.752 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.750+0000 7f175affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:09.752 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.750+0000 7f175affd640 10 monclient(hunting): my global_id is 4211 2024-03-25T18:04:09.752 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.750+0000 7f175affd640 1 -- 172.21.15.66:0/3007396598 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f174c004eb0 con 0x7f176c0b4d60 2024-03-25T18:04:09.752 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.750+0000 7f175affd640 1 -- 172.21.15.66:0/3007396598 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1808673526 0 0) 0x7f176c1d8540 con 0x7f176c0b1730 2024-03-25T18:04:09.752 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.750+0000 7f175affd640 10 monclient(hunting): _negotiate 2024-03-25T18:04:09.752 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.750+0000 7f175affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:09.752 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.750+0000 7f175affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:09.752 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.750+0000 7f175affd640 10 monclient(hunting): my global_id is 4210 2024-03-25T18:04:09.752 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.750+0000 7f175affd640 1 -- 172.21.15.66:0/3007396598 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f176c142d80 con 0x7f176c0b1730 2024-03-25T18:04:09.752 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.751+0000 7f175affd640 1 -- 172.21.15.66:0/3007396598 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3608624672 0 0) 0x7f176c13a960 con 0x7f176c14be90 2024-03-25T18:04:09.753 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.751+0000 7f175affd640 10 monclient(hunting): _negotiate 2024-03-25T18:04:09.753 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.751+0000 7f175affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:09.753 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.751+0000 7f175affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:09.753 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.751+0000 7f175affd640 10 monclient(hunting): my global_id is 4212 2024-03-25T18:04:09.753 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.751+0000 7f175affd640 1 -- 172.21.15.66:0/3007396598 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f174c005360 con 0x7f176c14be90 2024-03-25T18:04:09.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.756+0000 7f175affd640 1 -- 172.21.15.66:0/3007396598 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3373017792 0 0) 0x7f176c142d80 con 0x7f176c0b1730 2024-03-25T18:04:09.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.756+0000 7f175affd640 1 -- 172.21.15.66:0/3007396598 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f176c13a960 con 0x7f176c0b1730 2024-03-25T18:04:09.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.757+0000 7f175affd640 1 -- 172.21.15.66:0/3007396598 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f175c001ae0 con 0x7f176c0b1730 2024-03-25T18:04:09.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.757+0000 7f175affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:09.759 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.757+0000 7f175affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:04:09.759 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.757+0000 7f175affd640 10 monclient(hunting): dump: 2024-03-25T18:04:09.759 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:09.759 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:09.759 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:09.759 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:09.759 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:09.759 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:09.759 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:09.759 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:09.759 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:09.759 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:09.759 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.757+0000 7f175affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:04:09.760 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.757+0000 7f175affd640 1 -- 172.21.15.66:0/3007396598 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1410447521 0 0) 0x7f174c004eb0 con 0x7f176c0b4d60 2024-03-25T18:04:09.760 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.757+0000 7f175affd640 1 -- 172.21.15.66:0/3007396598 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f174c005c30 con 0x7f176c0b4d60 2024-03-25T18:04:09.760 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.757+0000 7f175affd640 1 -- 172.21.15.66:0/3007396598 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f1760001910 con 0x7f176c0b4d60 2024-03-25T18:04:09.760 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.757+0000 7f175affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:09.760 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.757+0000 7f175affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:04:09.760 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.757+0000 7f175affd640 10 monclient(hunting): dump: 2024-03-25T18:04:09.760 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:09.760 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:09.760 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:09.760 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:09.760 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:09.760 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:09.760 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:09.760 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:09.761 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:09.761 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:09.761 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.757+0000 7f175affd640 1 -- 172.21.15.66:0/3007396598 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1192225314 0 0) 0x7f174c005360 con 0x7f176c14be90 2024-03-25T18:04:09.761 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.757+0000 7f175affd640 1 -- 172.21.15.66:0/3007396598 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f174c004eb0 con 0x7f176c14be90 2024-03-25T18:04:09.761 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.757+0000 7f175affd640 1 -- 172.21.15.66:0/3007396598 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f1764001c30 con 0x7f176c14be90 2024-03-25T18:04:09.761 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.757+0000 7f175affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:09.761 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.757+0000 7f175affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:04:09.761 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.757+0000 7f175affd640 10 monclient(hunting): dump: 2024-03-25T18:04:09.761 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:09.761 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:09.761 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:09.761 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:09.761 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:09.761 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:09.761 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:09.761 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:09.761 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:09.762 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:09.764 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.761+0000 7f175affd640 1 -- 172.21.15.66:0/3007396598 <== mon.0 v1:172.21.15.66:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4128357546 0 0) 0x7f174c004eb0 con 0x7f176c14be90 2024-03-25T18:04:09.764 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.761+0000 7f175affd640 1 -- 172.21.15.66:0/3007396598 >> v1:172.21.15.66:6790/0 conn(0x7f176c0b4d60 legacy=0x7f176c139290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:09.764 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.761+0000 7f175affd640 1 -- 172.21.15.66:0/3007396598 >> v1:172.21.15.193:6789/0 conn(0x7f176c0b1730 legacy=0x7f176c1d7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:09.764 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.761+0000 7f175affd640 10 monclient: _finish_hunting 0 2024-03-25T18:04:09.764 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.761+0000 7f175affd640 1 monclient: found mon.a 2024-03-25T18:04:09.764 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.761+0000 7f175affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:09.764 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.761+0000 7f175affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:04:09.764 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.761+0000 7f175affd640 1 -- 172.21.15.66:0/3007396598 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f176c142f10 con 0x7f176c14be90 2024-03-25T18:04:09.764 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.761+0000 7f175affd640 10 monclient: _finish_auth 0 2024-03-25T18:04:09.764 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.761+0000 7f175affd640 10 monclient: _check_auth_tickets 2024-03-25T18:04:09.765 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.761+0000 7f175affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:09.765 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.761+0000 7f1770f10640 5 monclient: authenticate success, global_id 4212 2024-03-25T18:04:09.765 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.761+0000 7f1770f10640 10 monclient: _renew_subs 2024-03-25T18:04:09.765 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.761+0000 7f1770f10640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:04:09.765 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.761+0000 7f1770f10640 1 -- 172.21.15.66:0/3007396598 --> v1:172.21.15.66:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f176c1399a0 con 0x7f176c14be90 2024-03-25T18:04:09.765 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.761+0000 7f1770f10640 10 client.4212.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:04:09.765 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.761+0000 7f1770f10640 10 monclient: _renew_subs 2024-03-25T18:04:09.765 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.761+0000 7f1770f10640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:04:09.765 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.761+0000 7f1770f10640 1 -- 172.21.15.66:0/3007396598 --> v1:172.21.15.66:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f176c13b240 con 0x7f176c14be90 2024-03-25T18:04:09.765 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.761+0000 7f1770f10640 1 librados: init done 2024-03-25T18:04:09.765 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.763+0000 7f1770f10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:09.765 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.763+0000 7f1770f10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:09.765 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.763+0000 7f1770f10640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:04:09.765 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.763+0000 7f1770f10640 1 -- 172.21.15.66:0/3007396598 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f176c1401f0 con 0x7f176c14be90 2024-03-25T18:04:09.785 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:17.171 35228== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:09.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.790+0000 7f175affd640 1 -- 172.21.15.66:0/3007396598 <== mon.0 v1:172.21.15.66:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1764002040 con 0x7f176c14be90 2024-03-25T18:04:09.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.790+0000 7f175affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:09.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.790+0000 7f175affd640 1 -- 172.21.15.66:0/3007396598 <== mon.0 v1:172.21.15.66:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f1764003b30 con 0x7f176c14be90 2024-03-25T18:04:09.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.790+0000 7f175affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:09.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.790+0000 7f175affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:04:09.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.790+0000 7f175affd640 10 monclient: dump: 2024-03-25T18:04:09.791 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:09.791 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:09.791 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:09.791 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:09.791 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:09.791 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:09.791 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:09.791 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:09.791 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:09.791 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:09.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.790+0000 7f1768d99640 4 set_mon_vals no callback set 2024-03-25T18:04:09.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.791+0000 7f175affd640 1 -- 172.21.15.66:0/3007396598 <== mon.0 v1:172.21.15.66:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f1764003d50 con 0x7f176c14be90 2024-03-25T18:04:09.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.791+0000 7f176a59c640 10 monclient: get_auth_request con 0x7f174c06e660 auth_method 0 2024-03-25T18:04:09.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.792+0000 7f175affd640 10 client.4212.objecter ms_handle_connect 0x7f174c06e660 2024-03-25T18:04:09.796 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.796+0000 7f175affd640 1 -- 172.21.15.66:0/3007396598 <== mon.0 v1:172.21.15.66:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f1764054880 con 0x7f176c14be90 2024-03-25T18:04:09.796 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.796+0000 7f175affd640 10 client.4212.objecter ms_dispatch 0x7f176c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:04:09.797 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.796+0000 7f175affd640 3 client.4212.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:04:09.797 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.796+0000 7f175affd640 3 client.4212.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:04:09.797 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:09.796+0000 7f175affd640 20 client.4212.objecter dump_active .. 0 homeless 2024-03-25T18:04:09.871 INFO:tasks.ceph.osd.2.smithi066.stderr:==00:00:00:17.240 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:10.079 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:10.079+0000 7f175affd640 1 -- 172.21.15.66:0/3007396598 <== mon.0 v1:172.21.15.66:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1764050390 con 0x7f176c14be90 2024-03-25T18:04:10.079 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:10.079+0000 7f175affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:10.079 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:10.079+0000 7f175affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:04:10.348 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:10.348+0000 7f1770f10640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:10.348 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:10.348+0000 7f1770f10640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:10.348 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:10.348+0000 7f1770f10640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:04:10.348 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:10.348+0000 7f1770f10640 1 -- 172.21.15.66:0/3007396598 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f176c0627b0 con 0x7f176c14be90 2024-03-25T18:04:10.401 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:10.401+0000 7f175affd640 1 -- 172.21.15.66:0/3007396598 <== mon.0 v1:172.21.15.66:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f17640522c0 con 0x7f176c14be90 2024-03-25T18:04:10.401 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:10.401+0000 7f175affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:10.401 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:10.401+0000 7f175affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:04:10.402 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:04:10.402 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:04:10.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:10.403+0000 7f1770f10640 10 librados: watch_flush enter 2024-03-25T18:04:10.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:10.403+0000 7f1770f10640 10 librados: watch_flush exit 2024-03-25T18:04:10.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:10.403+0000 7f1770f10640 20 client.4212.objecter shutdown clearing up homeless session... 2024-03-25T18:04:10.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:10.403+0000 7f1770f10640 10 client.4212.objecter successfully canceled tick 2024-03-25T18:04:10.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:10.404+0000 7f1770f10640 1 -- 172.21.15.66:0/3007396598 >> v1:172.21.15.193:6800/2275676803 conn(0x7f174c06e660 legacy=0x7f174c070b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:10.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:10.404+0000 7f1770f10640 10 monclient: shutdown 2024-03-25T18:04:10.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:10.404+0000 7f1770f10640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:10.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:10.404+0000 7f1770f10640 1 -- 172.21.15.66:0/3007396598 >> v1:172.21.15.66:6789/0 conn(0x7f176c14be90 legacy=0x7f176c13b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:10.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:10.404+0000 7f1770f10640 1 -- 172.21.15.66:0/3007396598 shutdown_connections 2024-03-25T18:04:10.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:10.404+0000 7f1770f10640 1 -- 172.21.15.66:0/3007396598 >> 172.21.15.66:0/3007396598 conn(0x7f176c0617a0 msgr2=0x7f176c1374d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:10.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:10.404+0000 7f1770f10640 1 -- 172.21.15.66:0/3007396598 shutdown_connections 2024-03-25T18:04:10.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:10.404+0000 7f1770f10640 1 -- 172.21.15.66:0/3007396598 wait complete. 2024-03-25T18:04:10.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:10.404+0000 7f1770f10640 1 librados: shutdown 2024-03-25T18:04:10.417 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:04:14.474 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:21.860 35229== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:14.478 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:21.864 35229== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:14.972 INFO:tasks.ceph.osd.3.smithi066.stderr:==00:00:00:22.334 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:14.976 INFO:tasks.ceph.osd.3.smithi066.stderr:==00:00:00:22.338 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:15.188 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:22.568 35230== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:15.192 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:22.571 35230== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:15.201 INFO:tasks.ceph.osd.1.smithi066.stderr:==00:00:00:22.576 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:15.205 INFO:tasks.ceph.osd.1.smithi066.stderr:==00:00:00:22.581 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:15.268 INFO:tasks.ceph.osd.0.smithi066.stderr:==00:00:00:22.639 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:15.274 INFO:tasks.ceph.osd.0.smithi066.stderr:==00:00:00:22.645 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:15.488 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:22.878 35227== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:15.492 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:22.882 35227== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:15.745 INFO:tasks.ceph.osd.2.smithi066.stderr:==00:00:00:23.114 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:15.752 INFO:tasks.ceph.osd.2.smithi066.stderr:==00:00:00:23.121 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:16.072 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:23.458 35228== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:16.078 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:23.464 35228== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:17.225 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:04:17.331 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.330+0000 7f5f9f3fb640 10 monclient: get_monmap_and_config 2024-03-25T18:04:17.331 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.330+0000 7f5f9f3fb640 10 monclient: build_initial_monmap 2024-03-25T18:04:17.331 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.330+0000 7f5f9f3fb640 10 monclient: monmap: 2024-03-25T18:04:17.331 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:04:17.332 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:17.332 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:04:17.330656+0000 2024-03-25T18:04:17.332 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:04:17.330656+0000 2024-03-25T18:04:17.332 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:17.332 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:17.332 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:04:17.332 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:04:17.332 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:04:17.332 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:17.332 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.331+0000 7f5f9f3fb640 1 Processor -- start 2024-03-25T18:04:17.333 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.331+0000 7f5f9f3fb640 1 -- start start 2024-03-25T18:04:17.333 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.331+0000 7f5f9f3fb640 10 monclient: init 2024-03-25T18:04:17.334 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.332+0000 7f5f9f3fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:17.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.332+0000 7f5f9f3fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:04:17.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.332+0000 7f5f9f3fb640 10 monclient(hunting): picked mon.noname-c con 0x7f5f980b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T18:04:17.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.332+0000 7f5f9f3fb640 10 monclient(hunting): picked mon.noname-a con 0x7f5f980b1730 addr v1:172.21.15.66:6789/0 2024-03-25T18:04:17.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.332+0000 7f5f9f3fb640 10 monclient(hunting): picked mon.noname-b con 0x7f5f9814be90 addr v1:172.21.15.66:6790/0 2024-03-25T18:04:17.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.332+0000 7f5f9f3fb640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f980af940 con 0x7f5f980b1730 2024-03-25T18:04:17.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.333+0000 7f5f9f3fb640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f980b0a70 con 0x7f5f980b4d60 2024-03-25T18:04:17.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.333+0000 7f5f9f3fb640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f980b0d50 con 0x7f5f9814be90 2024-03-25T18:04:17.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.333+0000 7f5f9f3fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:17.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.333+0000 7f5f9f3fb640 10 monclient(hunting): authenticate will time out at 7006.392410s 2024-03-25T18:04:17.354 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.354+0000 7f5f9d998640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f5f9814be90 0x7f5f9814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:51424/0 (socket says 172.21.15.66:51424) 2024-03-25T18:04:17.354 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.354+0000 7f5f9d998640 1 -- 172.21.15.66:0/2951269093 learned_addr learned my addr 172.21.15.66:0/2951269093 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:04:17.371 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.371+0000 7f5f8ffff640 1 -- 172.21.15.66:0/2951269093 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021618227 0 0) 0x7f5f980b0d50 con 0x7f5f9814be90 2024-03-25T18:04:17.371 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.371+0000 7f5f8ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:04:17.371 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.371+0000 7f5f8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:17.372 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.371+0000 7f5f8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:17.372 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.371+0000 7f5f8ffff640 10 monclient(hunting): my global_id is 4214 2024-03-25T18:04:17.372 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.371+0000 7f5f8ffff640 1 -- 172.21.15.66:0/2951269093 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f84003520 con 0x7f5f9814be90 2024-03-25T18:04:17.375 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.374+0000 7f5f8ffff640 1 -- 172.21.15.66:0/2951269093 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565412259 0 0) 0x7f5f980af940 con 0x7f5f980b1730 2024-03-25T18:04:17.375 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.374+0000 7f5f8ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:04:17.375 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.374+0000 7f5f8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:17.375 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.374+0000 7f5f8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:17.375 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.374+0000 7f5f8ffff640 10 monclient(hunting): my global_id is 4215 2024-03-25T18:04:17.375 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.374+0000 7f5f8ffff640 1 -- 172.21.15.66:0/2951269093 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f980b0d50 con 0x7f5f980b1730 2024-03-25T18:04:17.378 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.377+0000 7f5f8ffff640 1 -- 172.21.15.66:0/2951269093 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1705500140 0 0) 0x7f5f84003520 con 0x7f5f9814be90 2024-03-25T18:04:17.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.378+0000 7f5f8ffff640 1 -- 172.21.15.66:0/2951269093 >> v1:172.21.15.193:6789/0 conn(0x7f5f980b4d60 legacy=0x7f5f9813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:17.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.378+0000 7f5f8ffff640 1 -- 172.21.15.66:0/2951269093 >> v1:172.21.15.66:6789/0 conn(0x7f5f980b1730 legacy=0x7f5f980b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:17.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.378+0000 7f5f8ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:04:17.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.378+0000 7f5f8ffff640 1 monclient: found mon.noname-b 2024-03-25T18:04:17.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.378+0000 7f5f8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:17.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.378+0000 7f5f8ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.66:6790/0 2024-03-25T18:04:17.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.378+0000 7f5f8ffff640 1 -- 172.21.15.66:0/2951269093 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f980b1030 con 0x7f5f9814be90 2024-03-25T18:04:17.380 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.378+0000 7f5f8ffff640 10 monclient: _finish_auth 0 2024-03-25T18:04:17.380 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.378+0000 7f5f8ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:04:17.380 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.378+0000 7f5f8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:17.380 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.378+0000 7f5f9f3fb640 5 monclient: authenticate success, global_id 4214 2024-03-25T18:04:17.380 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.378+0000 7f5f9f3fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:04:17.380 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.378+0000 7f5f8ffff640 1 -- 172.21.15.66:0/2951269093 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f5f900019d0 con 0x7f5f9814be90 2024-03-25T18:04:17.380 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.378+0000 7f5f8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:17.380 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.378+0000 7f5f8ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:04:17.380 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.378+0000 7f5f8ffff640 10 monclient: dump: 2024-03-25T18:04:17.380 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:17.380 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:17.381 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:17.381 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:17.381 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:17.381 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:17.381 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:17.381 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:17.381 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:17.381 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:17.381 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.378+0000 7f5f9f3fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:04:17.387 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.380+0000 7f5f8ffff640 1 -- 172.21.15.66:0/2951269093 <== mon.2 v1:172.21.15.66:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f90001c20 con 0x7f5f9814be90 2024-03-25T18:04:17.387 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.380+0000 7f5f8ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:17.387 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.380+0000 7f5f8ffff640 1 -- 172.21.15.66:0/2951269093 <== mon.2 v1:172.21.15.66:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f5f90001fb0 con 0x7f5f9814be90 2024-03-25T18:04:17.387 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.380+0000 7f5f8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:17.387 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.380+0000 7f5f8ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:04:17.387 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.380+0000 7f5f8ffff640 10 monclient: dump: 2024-03-25T18:04:17.387 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:17.388 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:17.388 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:17.388 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:17.388 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:17.388 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:17.388 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:17.388 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:17.388 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:17.388 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:17.388 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.380+0000 7f5f9f3fb640 10 monclient: get_monmap_and_config success 2024-03-25T18:04:17.389 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.380+0000 7f5f9f3fb640 4 set_mon_vals no callback set 2024-03-25T18:04:17.389 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.381+0000 7f5f9f3fb640 10 monclient: shutdown 2024-03-25T18:04:17.389 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.381+0000 7f5f9f3fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:17.389 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.381+0000 7f5f9f3fb640 1 -- 172.21.15.66:0/2951269093 >> v1:172.21.15.66:6790/0 conn(0x7f5f9814be90 legacy=0x7f5f9814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:17.389 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.381+0000 7f5f9f3fb640 1 -- 172.21.15.66:0/2951269093 shutdown_connections 2024-03-25T18:04:17.389 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.381+0000 7f5f9f3fb640 1 -- 172.21.15.66:0/2951269093 >> 172.21.15.66:0/2951269093 conn(0x7f5f980617a0 msgr2=0x7f5f98061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:17.389 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.381+0000 7f5f9f3fb640 1 -- 172.21.15.66:0/2951269093 shutdown_connections 2024-03-25T18:04:17.389 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.381+0000 7f5f9f3fb640 1 -- 172.21.15.66:0/2951269093 wait complete. 2024-03-25T18:04:17.389 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.382+0000 7f5f9f3fb640 10 monclient: build_initial_monmap 2024-03-25T18:04:17.389 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.382+0000 7f5f9f3fb640 10 monclient: monmap: 2024-03-25T18:04:17.389 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:04:17.390 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:04:17.390 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:04:17.390 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:04:17.390 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:17.390 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:04:17.390 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:04:17.390 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:04:17.390 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:04:17.390 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:17.390 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.382+0000 7f5f9f3fb640 1 librados: starting msgr at 2024-03-25T18:04:17.391 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.382+0000 7f5f9f3fb640 1 librados: starting objecter 2024-03-25T18:04:17.391 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.383+0000 7f5f9f3fb640 1 Processor -- start 2024-03-25T18:04:17.391 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.383+0000 7f5f9f3fb640 1 -- start start 2024-03-25T18:04:17.391 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.383+0000 7f5f9f3fb640 1 librados: setting wanted keys 2024-03-25T18:04:17.391 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.383+0000 7f5f9f3fb640 1 librados: calling monclient init 2024-03-25T18:04:17.391 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.383+0000 7f5f9f3fb640 10 monclient: init 2024-03-25T18:04:17.391 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.383+0000 7f5f9f3fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:17.391 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.383+0000 7f5f9f3fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:04:17.391 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.383+0000 7f5f9f3fb640 10 monclient(hunting): picked mon.noname-b con 0x7f5f980b1730 addr v1:172.21.15.193:6789/0 2024-03-25T18:04:17.391 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.384+0000 7f5f9f3fb640 10 monclient(hunting): picked mon.noname-a con 0x7f5f980b4d60 addr v1:172.21.15.66:6789/0 2024-03-25T18:04:17.391 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.384+0000 7f5f9f3fb640 10 monclient(hunting): picked mon.noname-c con 0x7f5f9814be90 addr v1:172.21.15.66:6790/0 2024-03-25T18:04:17.391 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.384+0000 7f5f9f3fb640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f98139ea0 con 0x7f5f980b4d60 2024-03-25T18:04:17.392 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.384+0000 7f5f9f3fb640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f981356d0 con 0x7f5f980b1730 2024-03-25T18:04:17.392 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.384+0000 7f5f9f3fb640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f98143600 con 0x7f5f9814be90 2024-03-25T18:04:17.392 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.384+0000 7f5f9f3fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:17.392 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.384+0000 7f5f9f3fb640 10 monclient(hunting): authenticate will time out at 7006.443463s 2024-03-25T18:04:17.392 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.384+0000 7f5f9d197640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f5f980b1730 0x7f5f9813b850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:40102/0 (socket says 172.21.15.66:40102) 2024-03-25T18:04:17.392 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.384+0000 7f5f9c996640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f5f980b4d60 0x7f5f981de430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:33032/0 (socket says 172.21.15.66:33032) 2024-03-25T18:04:17.392 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.384+0000 7f5f9c996640 1 -- 172.21.15.66:0/3363879921 learned_addr learned my addr 172.21.15.66:0/3363879921 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:04:17.392 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.385+0000 7f5f8dffb640 10 client.?.objecter ms_handle_connect 0x7f5f9814be90 2024-03-25T18:04:17.392 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.385+0000 7f5f8dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:17.392 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.385+0000 7f5f8dffb640 10 client.?.objecter ms_handle_connect 0x7f5f980b1730 2024-03-25T18:04:17.392 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.385+0000 7f5f8dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:17.393 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.385+0000 7f5f8dffb640 10 client.?.objecter ms_handle_connect 0x7f5f980b4d60 2024-03-25T18:04:17.393 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.385+0000 7f5f8dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:17.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.387+0000 7f5f8dffb640 1 -- 172.21.15.66:0/3363879921 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3400716561 0 0) 0x7f5f981356d0 con 0x7f5f980b1730 2024-03-25T18:04:17.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.387+0000 7f5f8dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:04:17.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.387+0000 7f5f8dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:17.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.388+0000 7f5f8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:17.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.388+0000 7f5f8dffb640 10 monclient(hunting): my global_id is 4216 2024-03-25T18:04:17.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.388+0000 7f5f8dffb640 1 -- 172.21.15.66:0/3363879921 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f7c004eb0 con 0x7f5f980b1730 2024-03-25T18:04:17.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.388+0000 7f5f8dffb640 1 -- 172.21.15.66:0/3363879921 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2082221883 0 0) 0x7f5f98143600 con 0x7f5f9814be90 2024-03-25T18:04:17.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.388+0000 7f5f8dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:04:17.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.388+0000 7f5f8dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:17.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.388+0000 7f5f8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:17.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.388+0000 7f5f8dffb640 10 monclient(hunting): my global_id is 4217 2024-03-25T18:04:17.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.388+0000 7f5f8dffb640 1 -- 172.21.15.66:0/3363879921 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f981356d0 con 0x7f5f9814be90 2024-03-25T18:04:17.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.388+0000 7f5f8dffb640 1 -- 172.21.15.66:0/3363879921 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3650104104 0 0) 0x7f5f98139ea0 con 0x7f5f980b4d60 2024-03-25T18:04:17.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.388+0000 7f5f8dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:04:17.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.388+0000 7f5f8dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:17.397 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.388+0000 7f5f8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:17.397 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.388+0000 7f5f8dffb640 10 monclient(hunting): my global_id is 4218 2024-03-25T18:04:17.397 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.388+0000 7f5f8dffb640 1 -- 172.21.15.66:0/3363879921 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f98143600 con 0x7f5f980b4d60 2024-03-25T18:04:17.397 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.393+0000 7f5f8dffb640 1 -- 172.21.15.66:0/3363879921 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 612326615 0 0) 0x7f5f981356d0 con 0x7f5f9814be90 2024-03-25T18:04:17.397 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.393+0000 7f5f8dffb640 1 -- 172.21.15.66:0/3363879921 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f98139ea0 con 0x7f5f9814be90 2024-03-25T18:04:17.397 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.393+0000 7f5f8dffb640 1 -- 172.21.15.66:0/3363879921 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f5f90001f30 con 0x7f5f9814be90 2024-03-25T18:04:17.397 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.393+0000 7f5f8dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:17.397 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.393+0000 7f5f8dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:04:17.397 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.393+0000 7f5f8dffb640 10 monclient(hunting): dump: 2024-03-25T18:04:17.398 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:17.398 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:17.398 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:17.398 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:17.398 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:17.398 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:17.398 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:17.398 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:17.398 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:17.398 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:17.398 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.393+0000 7f5f8dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:04:17.400 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.396+0000 7f5f8dffb640 1 -- 172.21.15.66:0/3363879921 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1804397431 0 0) 0x7f5f7c004eb0 con 0x7f5f980b1730 2024-03-25T18:04:17.400 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.396+0000 7f5f8dffb640 1 -- 172.21.15.66:0/3363879921 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f981356d0 con 0x7f5f980b1730 2024-03-25T18:04:17.400 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.396+0000 7f5f8dffb640 1 -- 172.21.15.66:0/3363879921 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f5f94001970 con 0x7f5f980b1730 2024-03-25T18:04:17.401 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.396+0000 7f5f8dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:17.401 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.396+0000 7f5f8dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:04:17.401 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.396+0000 7f5f8dffb640 10 monclient(hunting): dump: 2024-03-25T18:04:17.401 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:17.401 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:17.401 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:17.401 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:17.401 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:17.401 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:17.401 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:17.401 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:17.401 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:17.402 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:17.402 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.396+0000 7f5f8dffb640 1 -- 172.21.15.66:0/3363879921 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2931464702 0 0) 0x7f5f98143600 con 0x7f5f980b4d60 2024-03-25T18:04:17.402 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.396+0000 7f5f8dffb640 1 -- 172.21.15.66:0/3363879921 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f7c004eb0 con 0x7f5f980b4d60 2024-03-25T18:04:17.402 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.396+0000 7f5f8dffb640 1 -- 172.21.15.66:0/3363879921 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f5f88001930 con 0x7f5f980b4d60 2024-03-25T18:04:17.402 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.396+0000 7f5f8dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:17.402 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.396+0000 7f5f8dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:04:17.402 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.396+0000 7f5f8dffb640 10 monclient(hunting): dump: 2024-03-25T18:04:17.402 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:17.402 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:17.402 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:17.402 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:17.403 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:17.403 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:17.403 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:17.403 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:17.403 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:17.403 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:17.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.398+0000 7f5f8dffb640 1 -- 172.21.15.66:0/3363879921 <== mon.2 v1:172.21.15.66:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2340319336 0 0) 0x7f5f98139ea0 con 0x7f5f9814be90 2024-03-25T18:04:17.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.398+0000 7f5f8dffb640 1 -- 172.21.15.66:0/3363879921 >> v1:172.21.15.193:6789/0 conn(0x7f5f980b1730 legacy=0x7f5f9813b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:17.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.398+0000 7f5f8dffb640 1 -- 172.21.15.66:0/3363879921 >> v1:172.21.15.66:6789/0 conn(0x7f5f980b4d60 legacy=0x7f5f981de430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:17.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.398+0000 7f5f8dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:04:17.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.398+0000 7f5f8dffb640 1 monclient: found mon.c 2024-03-25T18:04:17.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.398+0000 7f5f8dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:17.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.398+0000 7f5f8dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:04:17.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.398+0000 7f5f8dffb640 1 -- 172.21.15.66:0/3363879921 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9813b370 con 0x7f5f9814be90 2024-03-25T18:04:17.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.398+0000 7f5f8dffb640 10 monclient: _finish_auth 0 2024-03-25T18:04:17.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.398+0000 7f5f8dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:04:17.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.398+0000 7f5f8dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:17.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.398+0000 7f5f9f3fb640 5 monclient: authenticate success, global_id 4217 2024-03-25T18:04:17.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.398+0000 7f5f9f3fb640 10 monclient: _renew_subs 2024-03-25T18:04:17.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.398+0000 7f5f9f3fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:04:17.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.398+0000 7f5f9f3fb640 1 -- 172.21.15.66:0/3363879921 --> v1:172.21.15.66:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f9813a0f0 con 0x7f5f9814be90 2024-03-25T18:04:17.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.398+0000 7f5f9f3fb640 10 client.4217.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:04:17.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.398+0000 7f5f9f3fb640 10 monclient: _renew_subs 2024-03-25T18:04:17.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.398+0000 7f5f9f3fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:04:17.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.398+0000 7f5f9f3fb640 1 -- 172.21.15.66:0/3363879921 --> v1:172.21.15.66:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f9813a6b0 con 0x7f5f9814be90 2024-03-25T18:04:17.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.398+0000 7f5f9f3fb640 1 librados: init done 2024-03-25T18:04:17.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.400+0000 7f5f9f3fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:17.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.400+0000 7f5f9f3fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:17.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.400+0000 7f5f9f3fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:04:17.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.400+0000 7f5f9f3fb640 1 -- 172.21.15.66:0/3363879921 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f980627b0 con 0x7f5f9814be90 2024-03-25T18:04:17.418 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.417+0000 7f5f8dffb640 1 -- 172.21.15.66:0/3363879921 <== mon.2 v1:172.21.15.66:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f90002090 con 0x7f5f9814be90 2024-03-25T18:04:17.418 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.417+0000 7f5f8dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:17.419 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.418+0000 7f5f8dffb640 1 -- 172.21.15.66:0/3363879921 <== mon.2 v1:172.21.15.66:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f5f90004fb0 con 0x7f5f9814be90 2024-03-25T18:04:17.419 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.418+0000 7f5f8f7fe640 4 set_mon_vals no callback set 2024-03-25T18:04:17.419 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.418+0000 7f5f8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:17.419 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.418+0000 7f5f8dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:04:17.419 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.418+0000 7f5f8dffb640 10 monclient: dump: 2024-03-25T18:04:17.419 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:17.419 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:17.419 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:17.419 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:17.419 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:17.420 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:17.420 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:17.420 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:17.420 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:17.420 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:17.420 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.419+0000 7f5f8dffb640 1 -- 172.21.15.66:0/3363879921 <== mon.2 v1:172.21.15.66:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f5f9001af80 con 0x7f5f9814be90 2024-03-25T18:04:17.421 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.419+0000 7f5f9d197640 10 monclient: get_auth_request con 0x7f5f7c06e0f0 auth_method 0 2024-03-25T18:04:17.421 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.420+0000 7f5f8dffb640 10 client.4217.objecter ms_handle_connect 0x7f5f7c06e0f0 2024-03-25T18:04:17.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.421+0000 7f5f8dffb640 1 -- 172.21.15.66:0/3363879921 <== mon.2 v1:172.21.15.66:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f5f90054a20 con 0x7f5f9814be90 2024-03-25T18:04:17.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.421+0000 7f5f8dffb640 10 client.4217.objecter ms_dispatch 0x7f5f98000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:04:17.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.421+0000 7f5f8dffb640 3 client.4217.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:04:17.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.421+0000 7f5f8dffb640 3 client.4217.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:04:17.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.421+0000 7f5f8dffb640 20 client.4217.objecter dump_active .. 0 homeless 2024-03-25T18:04:17.675 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.674+0000 7f5f8dffb640 1 -- 172.21.15.66:0/3363879921 <== mon.2 v1:172.21.15.66:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5f90050360 con 0x7f5f9814be90 2024-03-25T18:04:17.675 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.674+0000 7f5f8dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:17.675 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.674+0000 7f5f8dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:04:17.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.909+0000 7f5f9f3fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:17.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.909+0000 7f5f9f3fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:17.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.909+0000 7f5f9f3fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:04:17.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.909+0000 7f5f9f3fb640 1 -- 172.21.15.66:0/3363879921 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f98069f40 con 0x7f5f9814be90 2024-03-25T18:04:17.927 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.927+0000 7f5f8dffb640 1 -- 172.21.15.66:0/3363879921 <== mon.2 v1:172.21.15.66:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f5f90052460 con 0x7f5f9814be90 2024-03-25T18:04:17.927 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.927+0000 7f5f8dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:17.927 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.927+0000 7f5f8dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:04:17.928 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:04:17.928 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:04:17.930 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.929+0000 7f5f9f3fb640 10 librados: watch_flush enter 2024-03-25T18:04:17.930 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.929+0000 7f5f9f3fb640 10 librados: watch_flush exit 2024-03-25T18:04:17.930 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.929+0000 7f5f9f3fb640 20 client.4217.objecter shutdown clearing up homeless session... 2024-03-25T18:04:17.930 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.929+0000 7f5f9f3fb640 10 client.4217.objecter successfully canceled tick 2024-03-25T18:04:17.930 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.930+0000 7f5f9f3fb640 1 -- 172.21.15.66:0/3363879921 >> v1:172.21.15.193:6800/2275676803 conn(0x7f5f7c06e0f0 legacy=0x7f5f7c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:17.930 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.930+0000 7f5f9f3fb640 10 monclient: shutdown 2024-03-25T18:04:17.931 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.930+0000 7f5f9f3fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:17.931 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.930+0000 7f5f9f3fb640 1 -- 172.21.15.66:0/3363879921 >> v1:172.21.15.66:6790/0 conn(0x7f5f9814be90 legacy=0x7f5f981e7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:17.931 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.931+0000 7f5f9f3fb640 1 -- 172.21.15.66:0/3363879921 shutdown_connections 2024-03-25T18:04:17.932 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.931+0000 7f5f9f3fb640 1 -- 172.21.15.66:0/3363879921 >> 172.21.15.66:0/3363879921 conn(0x7f5f980617a0 msgr2=0x7f5f980b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:17.932 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.931+0000 7f5f9f3fb640 1 -- 172.21.15.66:0/3363879921 shutdown_connections 2024-03-25T18:04:17.932 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.931+0000 7f5f9f3fb640 1 -- 172.21.15.66:0/3363879921 wait complete. 2024-03-25T18:04:17.932 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:17.931+0000 7f5f9f3fb640 1 librados: shutdown 2024-03-25T18:04:17.944 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:04:18.732 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:00:26.117 35229== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:19.344 INFO:tasks.ceph.osd.3.smithi066.stderr:==00:00:00:26.706 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:19.434 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:00:26.813 35230== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:19.452 INFO:tasks.ceph.osd.1.smithi066.stderr:==00:00:00:26.827 35447== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:19.672 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:00:27.062 35227== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:19.673 INFO:tasks.ceph.osd.0.smithi066.stderr:==00:00:00:27.045 35448== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:20.078 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:00:27.464 35228== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:20.319 INFO:tasks.ceph.osd.2.smithi066.stderr:==00:00:00:27.688 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:04:24.752 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:04:24.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.873+0000 7efda8f04640 10 monclient: get_monmap_and_config 2024-03-25T18:04:24.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.874+0000 7efda8f04640 10 monclient: build_initial_monmap 2024-03-25T18:04:24.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.874+0000 7efda8f04640 10 monclient: monmap: 2024-03-25T18:04:24.874 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:04:24.874 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:24.874 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:04:24.874521+0000 2024-03-25T18:04:24.875 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:04:24.874521+0000 2024-03-25T18:04:24.875 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:24.875 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:24.875 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:04:24.875 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:04:24.875 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:04:24.875 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:24.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.874+0000 7efda8f04640 1 Processor -- start 2024-03-25T18:04:24.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.875+0000 7efda8f04640 1 -- start start 2024-03-25T18:04:24.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.875+0000 7efda8f04640 10 monclient: init 2024-03-25T18:04:24.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.876+0000 7efda8f04640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:24.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.876+0000 7efda8f04640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:04:24.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.876+0000 7efda8f04640 10 monclient(hunting): picked mon.noname-b con 0x7efda40a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:04:24.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.876+0000 7efda8f04640 10 monclient(hunting): picked mon.noname-a con 0x7efda413f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:04:24.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.876+0000 7efda8f04640 10 monclient(hunting): picked mon.noname-c con 0x7efda41430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:04:24.878 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.876+0000 7efda8f04640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda4142520 con 0x7efda413f710 2024-03-25T18:04:24.878 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.876+0000 7efda8f04640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda40b28a0 con 0x7efda41430c0 2024-03-25T18:04:24.878 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.877+0000 7efda8f04640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda40b2a30 con 0x7efda40a98f0 2024-03-25T18:04:24.878 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.877+0000 7efda259c640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7efda40a98f0 0x7efda40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:46692/0 (socket says 172.21.15.66:46692) 2024-03-25T18:04:24.878 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.877+0000 7efda259c640 1 -- 172.21.15.66:0/652043929 learned_addr learned my addr 172.21.15.66:0/652043929 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:04:24.878 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.877+0000 7efda8f04640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:24.878 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.877+0000 7efda8f04640 10 monclient(hunting): authenticate will time out at 7013.936368s 2024-03-25T18:04:24.881 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.881+0000 7efda159a640 1 -- 172.21.15.66:0/652043929 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992960145 0 0) 0x7efda4142520 con 0x7efda413f710 2024-03-25T18:04:24.881 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.881+0000 7efda159a640 10 monclient(hunting): _negotiate 2024-03-25T18:04:24.881 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.881+0000 7efda159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:24.881 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.881+0000 7efda159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:24.881 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.881+0000 7efda159a640 10 monclient(hunting): my global_id is 4221 2024-03-25T18:04:24.881 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.881+0000 7efda159a640 1 -- 172.21.15.66:0/652043929 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd8c003520 con 0x7efda413f710 2024-03-25T18:04:24.881 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.881+0000 7efda159a640 1 -- 172.21.15.66:0/652043929 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856917802 0 0) 0x7efda40b2a30 con 0x7efda40a98f0 2024-03-25T18:04:24.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.881+0000 7efda159a640 10 monclient(hunting): _negotiate 2024-03-25T18:04:24.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.881+0000 7efda159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:24.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.881+0000 7efda159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:24.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.881+0000 7efda159a640 10 monclient(hunting): my global_id is 4220 2024-03-25T18:04:24.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.881+0000 7efda159a640 1 -- 172.21.15.66:0/652043929 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efda4142520 con 0x7efda40a98f0 2024-03-25T18:04:24.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.881+0000 7efda159a640 1 -- 172.21.15.66:0/652043929 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4144746680 0 0) 0x7efda40b28a0 con 0x7efda41430c0 2024-03-25T18:04:24.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.881+0000 7efda159a640 10 monclient(hunting): _negotiate 2024-03-25T18:04:24.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.881+0000 7efda159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:24.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.881+0000 7efda159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:24.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.881+0000 7efda159a640 10 monclient(hunting): my global_id is 4219 2024-03-25T18:04:24.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.881+0000 7efda159a640 1 -- 172.21.15.66:0/652043929 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efda40b2a30 con 0x7efda41430c0 2024-03-25T18:04:24.886 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.886+0000 7efda159a640 1 -- 172.21.15.66:0/652043929 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2893936372 0 0) 0x7efd8c003520 con 0x7efda413f710 2024-03-25T18:04:24.886 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.886+0000 7efda159a640 1 -- 172.21.15.66:0/652043929 >> v1:172.21.15.66:6790/0 conn(0x7efda40a98f0 legacy=0x7efda40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:24.886 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.886+0000 7efda159a640 1 -- 172.21.15.66:0/652043929 >> v1:172.21.15.193:6789/0 conn(0x7efda41430c0 legacy=0x7efda414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:24.886 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.886+0000 7efda159a640 10 monclient: _finish_hunting 0 2024-03-25T18:04:24.887 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.886+0000 7efda159a640 1 monclient: found mon.noname-a 2024-03-25T18:04:24.887 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.886+0000 7efda159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:24.887 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.886+0000 7efda159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.66:6789/0 2024-03-25T18:04:24.887 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.886+0000 7efda159a640 1 -- 172.21.15.66:0/652043929 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda40b44c0 con 0x7efda413f710 2024-03-25T18:04:24.887 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.886+0000 7efda159a640 10 monclient: _finish_auth 0 2024-03-25T18:04:24.887 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.886+0000 7efda159a640 10 monclient: _check_auth_tickets 2024-03-25T18:04:24.887 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.886+0000 7efda159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:24.887 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.886+0000 7efda8f04640 5 monclient: authenticate success, global_id 4221 2024-03-25T18:04:24.887 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.886+0000 7efda8f04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:04:24.888 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.887+0000 7efda159a640 1 -- 172.21.15.66:0/652043929 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7efd98001650 con 0x7efda413f710 2024-03-25T18:04:24.888 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.887+0000 7efda159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:24.888 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.887+0000 7efda159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:04:24.888 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.887+0000 7efda159a640 10 monclient: dump: 2024-03-25T18:04:24.888 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:24.888 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:24.888 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:24.888 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:24.889 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:24.889 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:24.889 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:24.889 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:24.889 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:24.889 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:24.889 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.887+0000 7efda8f04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:04:24.890 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.889+0000 7efda159a640 1 -- 172.21.15.66:0/652043929 <== mon.0 v1:172.21.15.66:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd980018a0 con 0x7efda413f710 2024-03-25T18:04:24.890 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.889+0000 7efda159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:24.890 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.889+0000 7efda159a640 1 -- 172.21.15.66:0/652043929 <== mon.0 v1:172.21.15.66:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7efd98001650 con 0x7efda413f710 2024-03-25T18:04:24.890 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.889+0000 7efda159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:24.890 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.889+0000 7efda159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:04:24.891 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.889+0000 7efda159a640 10 monclient: dump: 2024-03-25T18:04:24.891 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:24.891 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:24.891 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:24.891 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:24.891 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:24.891 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:24.891 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:24.891 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:24.891 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:24.891 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:24.891 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.889+0000 7efda8f04640 10 monclient: get_monmap_and_config success 2024-03-25T18:04:24.891 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.889+0000 7efda8f04640 4 set_mon_vals no callback set 2024-03-25T18:04:24.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.890+0000 7efda8f04640 10 monclient: shutdown 2024-03-25T18:04:24.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.890+0000 7efda8f04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:24.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.890+0000 7efda8f04640 1 -- 172.21.15.66:0/652043929 >> v1:172.21.15.66:6789/0 conn(0x7efda413f710 legacy=0x7efda41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:24.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.890+0000 7efda8f04640 1 -- 172.21.15.66:0/652043929 shutdown_connections 2024-03-25T18:04:24.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.890+0000 7efda8f04640 1 -- 172.21.15.66:0/652043929 >> 172.21.15.66:0/652043929 conn(0x7efda40b7ea0 msgr2=0x7efda40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:24.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.891+0000 7efda8f04640 1 -- 172.21.15.66:0/652043929 shutdown_connections 2024-03-25T18:04:24.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.891+0000 7efda8f04640 1 -- 172.21.15.66:0/652043929 wait complete. 2024-03-25T18:04:24.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.892+0000 7efda8f04640 10 monclient: build_initial_monmap 2024-03-25T18:04:24.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.892+0000 7efda8f04640 10 monclient: monmap: 2024-03-25T18:04:24.893 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:04:24.893 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:04:24.893 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:04:24.894 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:04:24.894 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:24.894 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:04:24.894 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:04:24.894 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:04:24.894 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:04:24.894 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:24.894 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.892+0000 7efda8f04640 1 librados: starting msgr at 2024-03-25T18:04:24.894 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.892+0000 7efda8f04640 1 librados: starting objecter 2024-03-25T18:04:24.894 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.892+0000 7efda8f04640 1 Processor -- start 2024-03-25T18:04:24.894 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.892+0000 7efda8f04640 1 -- start start 2024-03-25T18:04:24.894 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.892+0000 7efda8f04640 1 librados: setting wanted keys 2024-03-25T18:04:24.894 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.892+0000 7efda8f04640 1 librados: calling monclient init 2024-03-25T18:04:24.895 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.892+0000 7efda8f04640 10 monclient: init 2024-03-25T18:04:24.896 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.893+0000 7efda8f04640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:24.896 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.893+0000 7efda8f04640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:04:24.896 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.893+0000 7efda8f04640 10 monclient(hunting): picked mon.noname-c con 0x7efda40a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:04:24.896 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.893+0000 7efda8f04640 10 monclient(hunting): picked mon.noname-a con 0x7efda41430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:04:24.896 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.893+0000 7efda8f04640 10 monclient(hunting): picked mon.noname-b con 0x7efda414e5b0 addr v1:172.21.15.193:6789/0 2024-03-25T18:04:24.896 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.893+0000 7efda8f04640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda40af7b0 con 0x7efda41430c0 2024-03-25T18:04:24.897 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.893+0000 7efda8f04640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda40b03c0 con 0x7efda414e5b0 2024-03-25T18:04:24.897 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.893+0000 7efda8f04640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda40b0550 con 0x7efda40a98f0 2024-03-25T18:04:24.897 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.893+0000 7efda8f04640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:24.897 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.893+0000 7efda8f04640 10 monclient(hunting): authenticate will time out at 7013.952995s 2024-03-25T18:04:24.897 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.894+0000 7efda2d9d640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7efda414e5b0 0x7efda41e1f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:35826/0 (socket says 172.21.15.66:35826) 2024-03-25T18:04:24.897 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.894+0000 7efda2d9d640 1 -- 172.21.15.66:0/599864692 learned_addr learned my addr 172.21.15.66:0/599864692 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:04:24.897 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.895+0000 7efd92ffd640 10 client.?.objecter ms_handle_connect 0x7efda414e5b0 2024-03-25T18:04:24.897 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.895+0000 7efd92ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:24.897 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.895+0000 7efd92ffd640 10 client.?.objecter ms_handle_connect 0x7efda40a98f0 2024-03-25T18:04:24.897 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.895+0000 7efd92ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:24.897 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.896+0000 7efd92ffd640 10 client.?.objecter ms_handle_connect 0x7efda41430c0 2024-03-25T18:04:24.897 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.896+0000 7efd92ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:24.900 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.898+0000 7efd92ffd640 1 -- 172.21.15.66:0/599864692 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858382846 0 0) 0x7efda40b03c0 con 0x7efda414e5b0 2024-03-25T18:04:24.900 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.898+0000 7efd92ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:04:24.900 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.898+0000 7efd92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:24.900 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.898+0000 7efd92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:24.900 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.898+0000 7efd92ffd640 10 monclient(hunting): my global_id is 4222 2024-03-25T18:04:24.900 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.898+0000 7efd92ffd640 1 -- 172.21.15.66:0/599864692 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd84004eb0 con 0x7efda414e5b0 2024-03-25T18:04:24.900 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.898+0000 7efd92ffd640 1 -- 172.21.15.66:0/599864692 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2528240606 0 0) 0x7efda40b0550 con 0x7efda40a98f0 2024-03-25T18:04:24.900 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.898+0000 7efd92ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:04:24.900 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.898+0000 7efd92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:24.901 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.898+0000 7efd92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:24.901 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.898+0000 7efd92ffd640 10 monclient(hunting): my global_id is 4223 2024-03-25T18:04:24.901 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.898+0000 7efd92ffd640 1 -- 172.21.15.66:0/599864692 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efda40b03c0 con 0x7efda40a98f0 2024-03-25T18:04:24.901 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.899+0000 7efd92ffd640 1 -- 172.21.15.66:0/599864692 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058567961 0 0) 0x7efda40af7b0 con 0x7efda41430c0 2024-03-25T18:04:24.901 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.899+0000 7efd92ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:04:24.901 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.899+0000 7efd92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:24.901 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.899+0000 7efd92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:24.901 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.899+0000 7efd92ffd640 10 monclient(hunting): my global_id is 4224 2024-03-25T18:04:24.901 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.899+0000 7efd92ffd640 1 -- 172.21.15.66:0/599864692 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efda40b0550 con 0x7efda41430c0 2024-03-25T18:04:24.907 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.905+0000 7efd92ffd640 1 -- 172.21.15.66:0/599864692 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 342826145 0 0) 0x7efda40b03c0 con 0x7efda40a98f0 2024-03-25T18:04:24.907 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.905+0000 7efd92ffd640 1 -- 172.21.15.66:0/599864692 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efda40af7b0 con 0x7efda40a98f0 2024-03-25T18:04:24.907 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.905+0000 7efd92ffd640 1 -- 172.21.15.66:0/599864692 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7efd94001930 con 0x7efda40a98f0 2024-03-25T18:04:24.907 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.905+0000 7efd92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:24.907 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.905+0000 7efd92ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:04:24.907 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.905+0000 7efd92ffd640 10 monclient(hunting): dump: 2024-03-25T18:04:24.907 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:24.907 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:24.907 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:24.907 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:24.908 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:24.908 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:24.908 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:24.908 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:24.908 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:24.908 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:24.908 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.905+0000 7efd92ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:04:24.908 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.905+0000 7efd92ffd640 1 -- 172.21.15.66:0/599864692 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2761207886 0 0) 0x7efd84004eb0 con 0x7efda414e5b0 2024-03-25T18:04:24.908 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.905+0000 7efd92ffd640 1 -- 172.21.15.66:0/599864692 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efda40b03c0 con 0x7efda414e5b0 2024-03-25T18:04:24.908 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.906+0000 7efd92ffd640 1 -- 172.21.15.66:0/599864692 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7efd9c001d90 con 0x7efda414e5b0 2024-03-25T18:04:24.908 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.906+0000 7efd92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:24.908 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.906+0000 7efd92ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:04:24.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.906+0000 7efd92ffd640 10 monclient(hunting): dump: 2024-03-25T18:04:24.909 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:24.909 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:24.909 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:24.909 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:24.909 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:24.909 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:24.909 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:24.909 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:24.909 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:24.909 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:24.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.906+0000 7efd92ffd640 1 -- 172.21.15.66:0/599864692 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 968517216 0 0) 0x7efda40b0550 con 0x7efda41430c0 2024-03-25T18:04:24.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.906+0000 7efd92ffd640 1 -- 172.21.15.66:0/599864692 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd84004eb0 con 0x7efda41430c0 2024-03-25T18:04:24.911 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.911+0000 7efd92ffd640 1 -- 172.21.15.66:0/599864692 <== mon.2 v1:172.21.15.66:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3502931756 0 0) 0x7efda40af7b0 con 0x7efda40a98f0 2024-03-25T18:04:24.911 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.911+0000 7efd92ffd640 1 -- 172.21.15.66:0/599864692 >> v1:172.21.15.193:6789/0 conn(0x7efda414e5b0 legacy=0x7efda41e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:24.911 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.911+0000 7efd92ffd640 1 -- 172.21.15.66:0/599864692 >> v1:172.21.15.66:6789/0 conn(0x7efda41430c0 legacy=0x7efda40afcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:24.911 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.911+0000 7efd92ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:04:24.911 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.911+0000 7efd92ffd640 1 monclient: found mon.c 2024-03-25T18:04:24.911 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.911+0000 7efd92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:24.911 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.911+0000 7efd92ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:04:24.912 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.911+0000 7efd92ffd640 1 -- 172.21.15.66:0/599864692 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda40b1a70 con 0x7efda40a98f0 2024-03-25T18:04:24.912 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.911+0000 7efd92ffd640 10 monclient: _finish_auth 0 2024-03-25T18:04:24.912 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.911+0000 7efd92ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:04:24.912 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.911+0000 7efd92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:24.912 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.911+0000 7efda8f04640 5 monclient: authenticate success, global_id 4223 2024-03-25T18:04:24.912 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.911+0000 7efda8f04640 10 monclient: _renew_subs 2024-03-25T18:04:24.912 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.911+0000 7efda8f04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:04:24.912 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.911+0000 7efda8f04640 1 -- 172.21.15.66:0/599864692 --> v1:172.21.15.66:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efda40b1ce0 con 0x7efda40a98f0 2024-03-25T18:04:24.912 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.911+0000 7efda8f04640 10 client.4223.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:04:24.912 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.911+0000 7efda8f04640 10 monclient: _renew_subs 2024-03-25T18:04:24.912 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.911+0000 7efda8f04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:04:24.912 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.911+0000 7efda8f04640 1 -- 172.21.15.66:0/599864692 --> v1:172.21.15.66:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efda41e3810 con 0x7efda40a98f0 2024-03-25T18:04:24.912 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.911+0000 7efda8f04640 1 librados: init done 2024-03-25T18:04:24.913 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.912+0000 7efda8f04640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:24.913 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.912+0000 7efda8f04640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:24.913 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.912+0000 7efda8f04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:04:24.913 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.912+0000 7efda8f04640 1 -- 172.21.15.66:0/599864692 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efda40627b0 con 0x7efda40a98f0 2024-03-25T18:04:24.942 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.942+0000 7efd92ffd640 1 -- 172.21.15.66:0/599864692 <== mon.2 v1:172.21.15.66:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd94001d40 con 0x7efda40a98f0 2024-03-25T18:04:24.942 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.942+0000 7efd92ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:24.942 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.942+0000 7efd92ffd640 1 -- 172.21.15.66:0/599864692 <== mon.2 v1:172.21.15.66:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7efd94004980 con 0x7efda40a98f0 2024-03-25T18:04:24.942 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.942+0000 7efd92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:24.942 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.942+0000 7efd92ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:04:24.942 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.942+0000 7efd92ffd640 10 monclient: dump: 2024-03-25T18:04:24.942 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:24.942 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:24.943 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:24.943 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:24.943 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:24.943 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:24.943 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:24.943 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:24.943 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:24.943 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:24.943 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.942+0000 7efda0d99640 4 set_mon_vals no callback set 2024-03-25T18:04:24.944 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.943+0000 7efd92ffd640 1 -- 172.21.15.66:0/599864692 <== mon.2 v1:172.21.15.66:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7efd9401a9d0 con 0x7efda40a98f0 2024-03-25T18:04:24.944 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.944+0000 7efda1d9b640 10 monclient: get_auth_request con 0x7efd8406e150 auth_method 0 2024-03-25T18:04:24.944 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.944+0000 7efd92ffd640 10 client.4223.objecter ms_handle_connect 0x7efd8406e150 2024-03-25T18:04:24.947 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.947+0000 7efd92ffd640 1 -- 172.21.15.66:0/599864692 <== mon.2 v1:172.21.15.66:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7efd94054660 con 0x7efda40a98f0 2024-03-25T18:04:24.947 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.947+0000 7efd92ffd640 10 client.4223.objecter ms_dispatch 0x7efda4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:04:24.948 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.947+0000 7efd92ffd640 3 client.4223.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:04:24.948 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.947+0000 7efd92ffd640 3 client.4223.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:04:24.948 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:24.947+0000 7efd92ffd640 20 client.4223.objecter dump_active .. 0 homeless 2024-03-25T18:04:25.325 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:25.324+0000 7efd92ffd640 1 -- 172.21.15.66:0/599864692 <== mon.2 v1:172.21.15.66:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efd9404ffa0 con 0x7efda40a98f0 2024-03-25T18:04:25.325 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:25.325+0000 7efd92ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:25.325 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:25.325+0000 7efd92ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:04:25.555 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:25.555+0000 7efda8f04640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:25.555 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:25.555+0000 7efda8f04640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:25.556 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:25.555+0000 7efda8f04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:04:25.556 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:25.555+0000 7efda8f04640 1 -- 172.21.15.66:0/599864692 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efda4074cc0 con 0x7efda40a98f0 2024-03-25T18:04:25.573 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:25.573+0000 7efd92ffd640 1 -- 172.21.15.66:0/599864692 <== mon.2 v1:172.21.15.66:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7efd940520a0 con 0x7efda40a98f0 2024-03-25T18:04:25.573 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:25.573+0000 7efd92ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:25.573 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:25.573+0000 7efd92ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:04:25.574 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:04:25.574 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:04:25.576 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:25.575+0000 7efda8f04640 10 librados: watch_flush enter 2024-03-25T18:04:25.576 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:25.575+0000 7efda8f04640 10 librados: watch_flush exit 2024-03-25T18:04:25.576 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:25.575+0000 7efda8f04640 20 client.4223.objecter shutdown clearing up homeless session... 2024-03-25T18:04:25.576 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:25.575+0000 7efda8f04640 10 client.4223.objecter successfully canceled tick 2024-03-25T18:04:25.577 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:25.575+0000 7efda8f04640 1 -- 172.21.15.66:0/599864692 >> v1:172.21.15.193:6800/2275676803 conn(0x7efd8406e150 legacy=0x7efd84070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:25.577 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:25.575+0000 7efda8f04640 10 monclient: shutdown 2024-03-25T18:04:25.577 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:25.575+0000 7efda8f04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:25.577 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:25.575+0000 7efda8f04640 1 -- 172.21.15.66:0/599864692 >> v1:172.21.15.66:6790/0 conn(0x7efda40a98f0 legacy=0x7efda40aef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:25.577 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:25.575+0000 7efda8f04640 1 -- 172.21.15.66:0/599864692 shutdown_connections 2024-03-25T18:04:25.577 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:25.575+0000 7efda8f04640 1 -- 172.21.15.66:0/599864692 >> 172.21.15.66:0/599864692 conn(0x7efda40b7ea0 msgr2=0x7efda413c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:25.577 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:25.575+0000 7efda8f04640 1 -- 172.21.15.66:0/599864692 shutdown_connections 2024-03-25T18:04:25.577 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:25.575+0000 7efda8f04640 1 -- 172.21.15.66:0/599864692 wait complete. 2024-03-25T18:04:25.577 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:25.576+0000 7efda8f04640 1 librados: shutdown 2024-03-25T18:04:25.587 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:04:27.871 INFO:tasks.ceph.osd.7.smithi193.stderr:2024-03-25T18:04:27.867+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T18:04:28.005 INFO:tasks.ceph.osd.6.smithi193.stderr:2024-03-25T18:04:27.978+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T18:04:28.395 INFO:tasks.ceph.osd.3.smithi066.stderr:2024-03-25T18:04:28.392+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T18:04:28.509 INFO:tasks.ceph.osd.1.smithi066.stderr:2024-03-25T18:04:28.505+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T18:04:28.825 INFO:tasks.ceph.osd.4.smithi193.stderr:2024-03-25T18:04:28.821+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T18:04:28.948 INFO:tasks.ceph.osd.0.smithi066.stderr:2024-03-25T18:04:28.944+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T18:04:29.110 INFO:tasks.ceph.osd.5.smithi193.stderr:2024-03-25T18:04:29.106+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T18:04:29.353 INFO:tasks.ceph.osd.2.smithi066.stderr:2024-03-25T18:04:29.349+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T18:04:32.393 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:04:32.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.497+0000 7f87a039d640 10 monclient: get_monmap_and_config 2024-03-25T18:04:32.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.497+0000 7f87a039d640 10 monclient: build_initial_monmap 2024-03-25T18:04:32.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.497+0000 7f87a039d640 10 monclient: monmap: 2024-03-25T18:04:32.497 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:04:32.497 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:32.497 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:04:32.497982+0000 2024-03-25T18:04:32.498 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:04:32.497982+0000 2024-03-25T18:04:32.498 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:32.498 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:32.498 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:04:32.498 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:04:32.498 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:04:32.498 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:32.499 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.498+0000 7f87a039d640 1 Processor -- start 2024-03-25T18:04:32.499 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.498+0000 7f87a039d640 1 -- start start 2024-03-25T18:04:32.499 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.498+0000 7f87a039d640 10 monclient: init 2024-03-25T18:04:32.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.498+0000 7f87a039d640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:32.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.498+0000 7f87a039d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:04:32.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.498+0000 7f87a039d640 10 monclient(hunting): picked mon.noname-c con 0x7f87980a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:04:32.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.498+0000 7f87a039d640 10 monclient(hunting): picked mon.noname-a con 0x7f879813f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:04:32.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.499+0000 7f87a039d640 10 monclient(hunting): picked mon.noname-b con 0x7f87981430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:04:32.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.499+0000 7f87a039d640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8798142520 con 0x7f879813f710 2024-03-25T18:04:32.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.499+0000 7f87a039d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87980b28a0 con 0x7f87980a98f0 2024-03-25T18:04:32.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.499+0000 7f87a039d640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87980b2a30 con 0x7f87981430c0 2024-03-25T18:04:32.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.499+0000 7f87a039d640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:32.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.499+0000 7f87a039d640 10 monclient(hunting): authenticate will time out at 7021.558182s 2024-03-25T18:04:32.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.516+0000 7f879d938640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f879813f710 0x7f87981429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:40556/0 (socket says 172.21.15.66:40556) 2024-03-25T18:04:32.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.516+0000 7f879d938640 1 -- 172.21.15.66:0/3139536768 learned_addr learned my addr 172.21.15.66:0/3139536768 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:04:32.531 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.531+0000 7f879d137640 1 -- 172.21.15.66:0/3139536768 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 796335694 0 0) 0x7f8798142520 con 0x7f879813f710 2024-03-25T18:04:32.531 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.531+0000 7f879d137640 10 monclient(hunting): _negotiate 2024-03-25T18:04:32.531 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.531+0000 7f879d137640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:32.531 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.531+0000 7f879d137640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:32.531 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.531+0000 7f879d137640 10 monclient(hunting): my global_id is 4251 2024-03-25T18:04:32.531 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.531+0000 7f879d137640 1 -- 172.21.15.66:0/3139536768 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8784003520 con 0x7f879813f710 2024-03-25T18:04:32.550 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.549+0000 7f879d137640 1 -- 172.21.15.66:0/3139536768 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3200199027 0 0) 0x7f8784003520 con 0x7f879813f710 2024-03-25T18:04:32.550 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.549+0000 7f879d137640 1 -- 172.21.15.66:0/3139536768 >> v1:172.21.15.66:6790/0 conn(0x7f87981430c0 legacy=0x7f879814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:32.550 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.549+0000 7f879d137640 1 -- 172.21.15.66:0/3139536768 >> v1:172.21.15.193:6789/0 conn(0x7f87980a98f0 legacy=0x7f87980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:32.550 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.549+0000 7f879d137640 10 monclient: _finish_hunting 0 2024-03-25T18:04:32.550 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.549+0000 7f879d137640 1 monclient: found mon.noname-a 2024-03-25T18:04:32.551 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.549+0000 7f879d137640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:32.551 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.549+0000 7f879d137640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.66:6789/0 2024-03-25T18:04:32.551 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.549+0000 7f879d137640 1 -- 172.21.15.66:0/3139536768 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87980b44c0 con 0x7f879813f710 2024-03-25T18:04:32.551 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.549+0000 7f879d137640 10 monclient: _finish_auth 0 2024-03-25T18:04:32.551 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.549+0000 7f879d137640 10 monclient: _check_auth_tickets 2024-03-25T18:04:32.551 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.549+0000 7f879d137640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:32.551 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.549+0000 7f879d137640 1 -- 172.21.15.66:0/3139536768 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8788001690 con 0x7f879813f710 2024-03-25T18:04:32.551 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.549+0000 7f87a039d640 5 monclient: authenticate success, global_id 4251 2024-03-25T18:04:32.551 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.549+0000 7f87a039d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:04:32.551 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.549+0000 7f879d137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:32.551 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.549+0000 7f879d137640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:04:32.551 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.549+0000 7f879d137640 10 monclient: dump: 2024-03-25T18:04:32.551 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:32.551 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:32.551 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:32.551 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:32.552 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:32.552 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:32.552 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:32.552 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:32.552 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:32.552 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:32.552 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.550+0000 7f87a039d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:04:32.570 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.570+0000 7f879d137640 1 -- 172.21.15.66:0/3139536768 <== mon.0 v1:172.21.15.66:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87880017f0 con 0x7f879813f710 2024-03-25T18:04:32.570 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.570+0000 7f879d137640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:32.570 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.570+0000 7f879d137640 1 -- 172.21.15.66:0/3139536768 <== mon.0 v1:172.21.15.66:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8788001c00 con 0x7f879813f710 2024-03-25T18:04:32.570 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.570+0000 7f87a039d640 10 monclient: get_monmap_and_config success 2024-03-25T18:04:32.571 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.570+0000 7f87a039d640 4 set_mon_vals no callback set 2024-03-25T18:04:32.571 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.570+0000 7f879d137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:32.571 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.570+0000 7f879d137640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:04:32.571 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.570+0000 7f879d137640 10 monclient: dump: 2024-03-25T18:04:32.571 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:32.571 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:32.571 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:32.571 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:32.571 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:32.571 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:32.571 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:32.571 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:32.571 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:32.572 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:32.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.571+0000 7f87a039d640 10 monclient: shutdown 2024-03-25T18:04:32.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.571+0000 7f87a039d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:32.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.571+0000 7f87a039d640 1 -- 172.21.15.66:0/3139536768 >> v1:172.21.15.66:6789/0 conn(0x7f879813f710 legacy=0x7f87981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:32.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.571+0000 7f87a039d640 1 -- 172.21.15.66:0/3139536768 shutdown_connections 2024-03-25T18:04:32.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.571+0000 7f87a039d640 1 -- 172.21.15.66:0/3139536768 >> 172.21.15.66:0/3139536768 conn(0x7f87980b7ea0 msgr2=0x7f87980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:32.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.571+0000 7f87a039d640 1 -- 172.21.15.66:0/3139536768 shutdown_connections 2024-03-25T18:04:32.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.571+0000 7f87a039d640 1 -- 172.21.15.66:0/3139536768 wait complete. 2024-03-25T18:04:32.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.572+0000 7f87a039d640 10 monclient: build_initial_monmap 2024-03-25T18:04:32.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.572+0000 7f87a039d640 10 monclient: monmap: 2024-03-25T18:04:32.575 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:04:32.575 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:04:32.575 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:04:32.575 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:04:32.575 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:32.575 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:04:32.575 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:04:32.575 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:04:32.575 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:04:32.575 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:32.575 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.572+0000 7f87a039d640 1 librados: starting msgr at 2024-03-25T18:04:32.575 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.572+0000 7f87a039d640 1 librados: starting objecter 2024-03-25T18:04:32.575 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.572+0000 7f87a039d640 1 Processor -- start 2024-03-25T18:04:32.575 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.572+0000 7f87a039d640 1 -- start start 2024-03-25T18:04:32.575 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.572+0000 7f87a039d640 1 librados: setting wanted keys 2024-03-25T18:04:32.576 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.572+0000 7f87a039d640 1 librados: calling monclient init 2024-03-25T18:04:32.576 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.572+0000 7f87a039d640 10 monclient: init 2024-03-25T18:04:32.576 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.573+0000 7f87a039d640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:32.576 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.573+0000 7f87a039d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:04:32.576 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.573+0000 7f87a039d640 10 monclient(hunting): picked mon.noname-c con 0x7f87980a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:04:32.576 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.573+0000 7f87a039d640 10 monclient(hunting): picked mon.noname-b con 0x7f879813f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:04:32.576 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.573+0000 7f87a039d640 10 monclient(hunting): picked mon.noname-a con 0x7f87981430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:04:32.576 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.573+0000 7f87a039d640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87980abd70 con 0x7f87981430c0 2024-03-25T18:04:32.576 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.573+0000 7f87a039d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87981d7710 con 0x7f879813f710 2024-03-25T18:04:32.576 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.573+0000 7f87a039d640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87981d8340 con 0x7f87980a98f0 2024-03-25T18:04:32.576 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.573+0000 7f87a039d640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:32.576 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.573+0000 7f87a039d640 10 monclient(hunting): authenticate will time out at 7021.632565s 2024-03-25T18:04:32.581 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.581+0000 7f879d938640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f879813f710 0x7f87981d7c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:37188/0 (socket says 172.21.15.66:37188) 2024-03-25T18:04:32.581 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.581+0000 7f879d938640 1 -- 172.21.15.66:0/1704486666 learned_addr learned my addr 172.21.15.66:0/1704486666 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:04:32.582 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.582+0000 7f878effd640 10 client.?.objecter ms_handle_connect 0x7f879813f710 2024-03-25T18:04:32.582 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.582+0000 7f878effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:32.585 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.585+0000 7f878effd640 1 -- 172.21.15.66:0/1704486666 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2967833930 0 0) 0x7f87981d7710 con 0x7f879813f710 2024-03-25T18:04:32.585 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.585+0000 7f878effd640 10 monclient(hunting): _negotiate 2024-03-25T18:04:32.585 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.585+0000 7f878effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:32.585 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.585+0000 7f878effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:32.585 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.585+0000 7f878effd640 10 monclient(hunting): my global_id is 4252 2024-03-25T18:04:32.586 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.585+0000 7f878effd640 1 -- 172.21.15.66:0/1704486666 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f877c004eb0 con 0x7f879813f710 2024-03-25T18:04:32.590 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.590+0000 7f878effd640 10 client.?.objecter ms_handle_connect 0x7f87980a98f0 2024-03-25T18:04:32.590 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.590+0000 7f878effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:32.596 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.596+0000 7f878effd640 1 -- 172.21.15.66:0/1704486666 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 250804879 0 0) 0x7f87981d8340 con 0x7f87980a98f0 2024-03-25T18:04:32.596 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.596+0000 7f878effd640 10 monclient(hunting): _negotiate 2024-03-25T18:04:32.596 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.596+0000 7f878effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:32.596 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.596+0000 7f878effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:32.597 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.596+0000 7f878effd640 10 monclient(hunting): my global_id is 4253 2024-03-25T18:04:32.597 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.596+0000 7f878effd640 1 -- 172.21.15.66:0/1704486666 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87981d7710 con 0x7f87980a98f0 2024-03-25T18:04:32.601 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.601+0000 7f878effd640 10 client.?.objecter ms_handle_connect 0x7f87981430c0 2024-03-25T18:04:32.602 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.601+0000 7f878effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:32.606 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.606+0000 7f878effd640 1 -- 172.21.15.66:0/1704486666 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2323478521 0 0) 0x7f87981d7710 con 0x7f87980a98f0 2024-03-25T18:04:32.606 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.606+0000 7f878effd640 1 -- 172.21.15.66:0/1704486666 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87981d8340 con 0x7f87980a98f0 2024-03-25T18:04:32.607 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.606+0000 7f878effd640 1 -- 172.21.15.66:0/1704486666 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8794001930 con 0x7f87980a98f0 2024-03-25T18:04:32.607 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.606+0000 7f878effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:32.607 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.606+0000 7f878effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:04:32.607 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.606+0000 7f878effd640 10 monclient(hunting): dump: 2024-03-25T18:04:32.607 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:32.607 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:32.607 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:32.607 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:32.607 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:32.607 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:32.607 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:32.607 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:32.607 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:32.607 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:32.608 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.606+0000 7f878effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:04:32.614 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.612+0000 7f878effd640 1 -- 172.21.15.66:0/1704486666 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1023968718 0 0) 0x7f877c004eb0 con 0x7f879813f710 2024-03-25T18:04:32.614 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.612+0000 7f878effd640 1 -- 172.21.15.66:0/1704486666 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87981d7710 con 0x7f879813f710 2024-03-25T18:04:32.614 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.612+0000 7f878effd640 1 -- 172.21.15.66:0/1704486666 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8788001bc0 con 0x7f879813f710 2024-03-25T18:04:32.614 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.612+0000 7f878effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:32.614 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.612+0000 7f878effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:04:32.614 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.612+0000 7f878effd640 10 monclient(hunting): dump: 2024-03-25T18:04:32.614 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:32.614 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:32.614 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:32.614 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:32.614 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:32.614 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:32.615 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:32.615 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:32.615 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:32.615 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:32.615 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.612+0000 7f878effd640 1 -- 172.21.15.66:0/1704486666 <== mon.2 v1:172.21.15.66:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1410829657 0 0) 0x7f87981d8340 con 0x7f87980a98f0 2024-03-25T18:04:32.615 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.612+0000 7f878effd640 1 -- 172.21.15.66:0/1704486666 >> v1:172.21.15.193:6789/0 conn(0x7f879813f710 legacy=0x7f87981d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:32.615 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.612+0000 7f878effd640 1 -- 172.21.15.66:0/1704486666 >> v1:172.21.15.66:6789/0 conn(0x7f87981430c0 legacy=0x7f87981d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:32.615 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.612+0000 7f878effd640 10 monclient: _finish_hunting 0 2024-03-25T18:04:32.615 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.612+0000 7f878effd640 1 monclient: found mon.c 2024-03-25T18:04:32.615 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.612+0000 7f878effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:32.615 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.612+0000 7f878effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:04:32.615 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.612+0000 7f878effd640 1 -- 172.21.15.66:0/1704486666 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87981d84d0 con 0x7f87980a98f0 2024-03-25T18:04:32.615 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.612+0000 7f878effd640 10 monclient: _finish_auth 0 2024-03-25T18:04:32.615 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.612+0000 7f878effd640 10 monclient: _check_auth_tickets 2024-03-25T18:04:32.615 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.612+0000 7f878effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:32.616 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.613+0000 7f87a039d640 5 monclient: authenticate success, global_id 4253 2024-03-25T18:04:32.616 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.613+0000 7f87a039d640 10 monclient: _renew_subs 2024-03-25T18:04:32.616 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.613+0000 7f87a039d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:04:32.616 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.613+0000 7f87a039d640 1 -- 172.21.15.66:0/1704486666 --> v1:172.21.15.66:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87981d41e0 con 0x7f87980a98f0 2024-03-25T18:04:32.616 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.613+0000 7f87a039d640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:04:32.616 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.613+0000 7f87a039d640 10 monclient: _renew_subs 2024-03-25T18:04:32.616 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.613+0000 7f87a039d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:04:32.616 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.613+0000 7f87a039d640 1 -- 172.21.15.66:0/1704486666 --> v1:172.21.15.66:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f87981d5150 con 0x7f87980a98f0 2024-03-25T18:04:32.616 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.613+0000 7f87a039d640 1 librados: init done 2024-03-25T18:04:32.616 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.614+0000 7f87a039d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:32.616 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.614+0000 7f87a039d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:32.616 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.614+0000 7f87a039d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:04:32.616 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.614+0000 7f87a039d640 1 -- 172.21.15.66:0/1704486666 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87980627b0 con 0x7f87980a98f0 2024-03-25T18:04:32.937 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.937+0000 7f878effd640 1 -- 172.21.15.66:0/1704486666 <== mon.2 v1:172.21.15.66:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8794003610 con 0x7f87980a98f0 2024-03-25T18:04:32.938 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.937+0000 7f878effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:32.938 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.937+0000 7f878effd640 1 -- 172.21.15.66:0/1704486666 <== mon.2 v1:172.21.15.66:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8794003960 con 0x7f87980a98f0 2024-03-25T18:04:32.938 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.937+0000 7f878effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:32.938 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.937+0000 7f878effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:04:32.938 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.937+0000 7f878effd640 10 monclient: dump: 2024-03-25T18:04:32.938 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:32.938 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:32.938 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:32.938 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:32.938 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:32.938 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:32.938 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:32.938 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:32.938 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:32.939 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:32.939 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.937+0000 7f879c936640 4 set_mon_vals no callback set 2024-03-25T18:04:32.939 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.938+0000 7f878effd640 1 -- 172.21.15.66:0/1704486666 <== mon.2 v1:172.21.15.66:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f8794003b80 con 0x7f87980a98f0 2024-03-25T18:04:32.939 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.939+0000 7f879d938640 10 monclient: get_auth_request con 0x7f877c06de90 auth_method 0 2024-03-25T18:04:32.939 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.939+0000 7f878effd640 10 client.4253.objecter ms_handle_connect 0x7f877c06de90 2024-03-25T18:04:32.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.945+0000 7f878effd640 1 -- 172.21.15.66:0/1704486666 <== mon.2 v1:172.21.15.66:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f8794053520 con 0x7f87980a98f0 2024-03-25T18:04:32.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.945+0000 7f878effd640 10 client.4253.objecter ms_dispatch 0x7f8798000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:04:32.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.945+0000 7f878effd640 3 client.4253.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:04:32.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.945+0000 7f878effd640 3 client.4253.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:04:32.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.945+0000 7f878effd640 20 client.4253.objecter dump_active .. 0 homeless 2024-03-25T18:04:32.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.945+0000 7f878effd640 1 -- 172.21.15.66:0/1704486666 <== mon.2 v1:172.21.15.66:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f879404f030 con 0x7f87980a98f0 2024-03-25T18:04:32.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.945+0000 7f878effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:32.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:32.945+0000 7f878effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:04:33.208 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:33.208+0000 7f87a039d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:33.208 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:33.208+0000 7f87a039d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:33.208 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:33.208+0000 7f87a039d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:04:33.208 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:33.208+0000 7f87a039d640 1 -- 172.21.15.66:0/1704486666 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8798074cc0 con 0x7f87980a98f0 2024-03-25T18:04:33.239 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:33.239+0000 7f878effd640 1 -- 172.21.15.66:0/1704486666 <== mon.2 v1:172.21.15.66:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f8794050f60 con 0x7f87980a98f0 2024-03-25T18:04:33.239 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:33.239+0000 7f878effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:33.239 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:33.239+0000 7f878effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:04:33.240 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:04:33.240 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:04:33.242 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:33.241+0000 7f87a039d640 10 librados: watch_flush enter 2024-03-25T18:04:33.242 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:33.242+0000 7f87a039d640 10 librados: watch_flush exit 2024-03-25T18:04:33.242 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:33.242+0000 7f87a039d640 20 client.4253.objecter shutdown clearing up homeless session... 2024-03-25T18:04:33.242 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:33.242+0000 7f87a039d640 10 client.4253.objecter successfully canceled tick 2024-03-25T18:04:33.243 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:33.242+0000 7f87a039d640 1 -- 172.21.15.66:0/1704486666 >> v1:172.21.15.193:6800/2275676803 conn(0x7f877c06de90 legacy=0x7f877c070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:33.243 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:33.242+0000 7f87a039d640 10 monclient: shutdown 2024-03-25T18:04:33.243 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:33.242+0000 7f87a039d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:33.243 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:33.242+0000 7f87a039d640 1 -- 172.21.15.66:0/1704486666 >> v1:172.21.15.66:6790/0 conn(0x7f87980a98f0 legacy=0x7f8798137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:33.243 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:33.242+0000 7f87a039d640 1 -- 172.21.15.66:0/1704486666 shutdown_connections 2024-03-25T18:04:33.243 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:33.242+0000 7f87a039d640 1 -- 172.21.15.66:0/1704486666 >> 172.21.15.66:0/1704486666 conn(0x7f87980b7ea0 msgr2=0x7f8798074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:33.243 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:33.242+0000 7f87a039d640 1 -- 172.21.15.66:0/1704486666 shutdown_connections 2024-03-25T18:04:33.243 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:33.242+0000 7f87a039d640 1 -- 172.21.15.66:0/1704486666 wait complete. 2024-03-25T18:04:33.243 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:33.242+0000 7f87a039d640 1 librados: shutdown 2024-03-25T18:04:33.253 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:04:40.060 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:04:40.179 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.177+0000 7ff1a3e2a640 10 monclient: get_monmap_and_config 2024-03-25T18:04:40.179 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.177+0000 7ff1a3e2a640 10 monclient: build_initial_monmap 2024-03-25T18:04:40.179 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.177+0000 7ff1a3e2a640 10 monclient: monmap: 2024-03-25T18:04:40.179 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:04:40.179 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:40.179 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:04:40.179055+0000 2024-03-25T18:04:40.179 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:04:40.179055+0000 2024-03-25T18:04:40.179 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:40.179 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:40.179 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:04:40.179 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:04:40.179 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:04:40.180 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:40.180 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.178+0000 7ff1a3e2a640 1 Processor -- start 2024-03-25T18:04:40.181 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.178+0000 7ff1a3e2a640 1 -- start start 2024-03-25T18:04:40.181 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.178+0000 7ff1a3e2a640 10 monclient: init 2024-03-25T18:04:40.181 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.179+0000 7ff1a3e2a640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:40.181 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.179+0000 7ff1a3e2a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:04:40.181 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.179+0000 7ff1a3e2a640 10 monclient(hunting): picked mon.noname-a con 0x7ff19c0a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:04:40.181 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.179+0000 7ff1a3e2a640 10 monclient(hunting): picked mon.noname-c con 0x7ff19c13f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:04:40.181 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.179+0000 7ff1a3e2a640 10 monclient(hunting): picked mon.noname-b con 0x7ff19c1430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:04:40.181 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.179+0000 7ff1a3e2a640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19c142520 con 0x7ff19c0a98f0 2024-03-25T18:04:40.182 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.179+0000 7ff1a3e2a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19c0b28a0 con 0x7ff19c13f710 2024-03-25T18:04:40.182 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.179+0000 7ff1a3e2a640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19c0b2a30 con 0x7ff19c1430c0 2024-03-25T18:04:40.182 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.179+0000 7ff1a3e2a640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:40.182 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.179+0000 7ff1a3e2a640 10 monclient(hunting): authenticate will time out at 7029.239797s 2024-03-25T18:04:40.183 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.182+0000 7ff1a23c7640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7ff19c1430c0 0x7ff19c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:49068/0 (socket says 172.21.15.66:49068) 2024-03-25T18:04:40.183 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.182+0000 7ff1a23c7640 1 -- 172.21.15.66:0/3221143462 learned_addr learned my addr 172.21.15.66:0/3221143462 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:04:40.192 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.191+0000 7ff1a0bc4640 1 -- 172.21.15.66:0/3221143462 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 433937079 0 0) 0x7ff19c0b2a30 con 0x7ff19c1430c0 2024-03-25T18:04:40.192 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.191+0000 7ff1a0bc4640 10 monclient(hunting): _negotiate 2024-03-25T18:04:40.192 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.191+0000 7ff1a0bc4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:40.192 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.191+0000 7ff1a0bc4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:40.192 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.191+0000 7ff1a0bc4640 10 monclient(hunting): my global_id is 4256 2024-03-25T18:04:40.192 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.191+0000 7ff1a0bc4640 1 -- 172.21.15.66:0/3221143462 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff188003520 con 0x7ff19c1430c0 2024-03-25T18:04:40.198 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.197+0000 7ff1a0bc4640 1 -- 172.21.15.66:0/3221143462 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3631346945 0 0) 0x7ff19c142520 con 0x7ff19c0a98f0 2024-03-25T18:04:40.198 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.197+0000 7ff1a0bc4640 10 monclient(hunting): _negotiate 2024-03-25T18:04:40.198 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.197+0000 7ff1a0bc4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:40.198 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.197+0000 7ff1a0bc4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:40.198 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.197+0000 7ff1a0bc4640 10 monclient(hunting): my global_id is 4257 2024-03-25T18:04:40.198 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.197+0000 7ff1a0bc4640 1 -- 172.21.15.66:0/3221143462 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff19c0b2a30 con 0x7ff19c0a98f0 2024-03-25T18:04:40.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.197+0000 7ff1a0bc4640 1 -- 172.21.15.66:0/3221143462 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 70607967 0 0) 0x7ff19c0b28a0 con 0x7ff19c13f710 2024-03-25T18:04:40.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.197+0000 7ff1a0bc4640 10 monclient(hunting): _negotiate 2024-03-25T18:04:40.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.197+0000 7ff1a0bc4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:40.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.197+0000 7ff1a0bc4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:40.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.197+0000 7ff1a0bc4640 10 monclient(hunting): my global_id is 4255 2024-03-25T18:04:40.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.197+0000 7ff1a0bc4640 1 -- 172.21.15.66:0/3221143462 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff19c142520 con 0x7ff19c13f710 2024-03-25T18:04:40.206 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.204+0000 7ff1a0bc4640 1 -- 172.21.15.66:0/3221143462 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 585679237 0 0) 0x7ff188003520 con 0x7ff19c1430c0 2024-03-25T18:04:40.206 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.204+0000 7ff1a0bc4640 1 -- 172.21.15.66:0/3221143462 >> v1:172.21.15.193:6789/0 conn(0x7ff19c13f710 legacy=0x7ff19c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:40.206 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.204+0000 7ff1a0bc4640 1 -- 172.21.15.66:0/3221143462 >> v1:172.21.15.66:6789/0 conn(0x7ff19c0a98f0 legacy=0x7ff19c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:40.206 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.204+0000 7ff1a0bc4640 10 monclient: _finish_hunting 0 2024-03-25T18:04:40.206 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.204+0000 7ff1a0bc4640 1 monclient: found mon.noname-b 2024-03-25T18:04:40.206 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.204+0000 7ff1a0bc4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:40.206 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.204+0000 7ff1a0bc4640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.66:6790/0 2024-03-25T18:04:40.206 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.204+0000 7ff1a0bc4640 1 -- 172.21.15.66:0/3221143462 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff19c0b44c0 con 0x7ff19c1430c0 2024-03-25T18:04:40.206 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.204+0000 7ff1a0bc4640 10 monclient: _finish_auth 0 2024-03-25T18:04:40.206 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.204+0000 7ff1a0bc4640 10 monclient: _check_auth_tickets 2024-03-25T18:04:40.206 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.204+0000 7ff1a0bc4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:40.206 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.204+0000 7ff1a0bc4640 1 -- 172.21.15.66:0/3221143462 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7ff194001a10 con 0x7ff19c1430c0 2024-03-25T18:04:40.207 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.204+0000 7ff1a0bc4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:40.207 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.204+0000 7ff1a0bc4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:04:40.207 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.204+0000 7ff1a0bc4640 10 monclient: dump: 2024-03-25T18:04:40.207 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:40.207 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:40.207 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:40.207 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:40.207 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:40.207 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:40.207 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:40.207 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:40.207 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:40.207 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:40.207 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.204+0000 7ff1a3e2a640 5 monclient: authenticate success, global_id 4256 2024-03-25T18:04:40.207 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.204+0000 7ff1a3e2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:04:40.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.231+0000 7ff1a0bc4640 1 -- 172.21.15.66:0/3221143462 <== mon.2 v1:172.21.15.66:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff194001b70 con 0x7ff19c1430c0 2024-03-25T18:04:40.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.232+0000 7ff1a0bc4640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:40.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.232+0000 7ff1a0bc4640 1 -- 172.21.15.66:0/3221143462 <== mon.2 v1:172.21.15.66:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7ff194001f80 con 0x7ff19c1430c0 2024-03-25T18:04:40.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.232+0000 7ff1a3e2a640 10 monclient: get_monmap_and_config success 2024-03-25T18:04:40.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.232+0000 7ff1a3e2a640 4 set_mon_vals no callback set 2024-03-25T18:04:40.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.232+0000 7ff1a0bc4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:40.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.232+0000 7ff1a0bc4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:04:40.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.232+0000 7ff1a0bc4640 10 monclient: dump: 2024-03-25T18:04:40.233 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:40.234 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:40.234 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:40.234 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:40.234 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:40.234 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:40.234 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:40.234 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:40.234 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:40.234 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:40.236 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.232+0000 7ff1a3e2a640 10 monclient: shutdown 2024-03-25T18:04:40.236 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.232+0000 7ff1a3e2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:40.236 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.232+0000 7ff1a3e2a640 1 -- 172.21.15.66:0/3221143462 >> v1:172.21.15.66:6790/0 conn(0x7ff19c1430c0 legacy=0x7ff19c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:40.236 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.232+0000 7ff1a3e2a640 1 -- 172.21.15.66:0/3221143462 shutdown_connections 2024-03-25T18:04:40.236 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.232+0000 7ff1a3e2a640 1 -- 172.21.15.66:0/3221143462 >> 172.21.15.66:0/3221143462 conn(0x7ff19c0b7ea0 msgr2=0x7ff19c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:40.236 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.233+0000 7ff1a3e2a640 1 -- 172.21.15.66:0/3221143462 shutdown_connections 2024-03-25T18:04:40.236 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.233+0000 7ff1a3e2a640 1 -- 172.21.15.66:0/3221143462 wait complete. 2024-03-25T18:04:40.236 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.233+0000 7ff1a3e2a640 10 monclient: build_initial_monmap 2024-03-25T18:04:40.236 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.233+0000 7ff1a3e2a640 10 monclient: monmap: 2024-03-25T18:04:40.236 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:04:40.237 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:04:40.237 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:04:40.237 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:04:40.237 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:40.237 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:04:40.237 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:04:40.237 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:04:40.237 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:04:40.237 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:40.237 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.234+0000 7ff1a3e2a640 1 librados: starting msgr at 2024-03-25T18:04:40.237 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.234+0000 7ff1a3e2a640 1 librados: starting objecter 2024-03-25T18:04:40.237 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.234+0000 7ff1a3e2a640 1 Processor -- start 2024-03-25T18:04:40.237 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.234+0000 7ff1a3e2a640 1 -- start start 2024-03-25T18:04:40.238 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.234+0000 7ff1a3e2a640 1 librados: setting wanted keys 2024-03-25T18:04:40.238 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.234+0000 7ff1a3e2a640 1 librados: calling monclient init 2024-03-25T18:04:40.238 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.234+0000 7ff1a3e2a640 10 monclient: init 2024-03-25T18:04:40.238 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.234+0000 7ff1a3e2a640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:40.238 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.234+0000 7ff1a3e2a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:04:40.238 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.234+0000 7ff1a3e2a640 10 monclient(hunting): picked mon.noname-a con 0x7ff19c0a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:04:40.238 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.234+0000 7ff1a3e2a640 10 monclient(hunting): picked mon.noname-b con 0x7ff19c13f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:04:40.238 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.234+0000 7ff1a3e2a640 10 monclient(hunting): picked mon.noname-c con 0x7ff19c1430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:04:40.238 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.234+0000 7ff1a3e2a640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19c136ad0 con 0x7ff19c0a98f0 2024-03-25T18:04:40.238 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.234+0000 7ff1a3e2a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19c0b7520 con 0x7ff19c13f710 2024-03-25T18:04:40.238 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.234+0000 7ff1a3e2a640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19c135810 con 0x7ff19c1430c0 2024-03-25T18:04:40.238 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.234+0000 7ff1a3e2a640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:40.238 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.234+0000 7ff1a3e2a640 10 monclient(hunting): authenticate will time out at 7029.294997s 2024-03-25T18:04:40.249 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.248+0000 7ff1a13c5640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7ff19c13f710 0x7ff19c0b5a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:60280/0 (socket says 172.21.15.66:60280) 2024-03-25T18:04:40.249 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.248+0000 7ff1a13c5640 1 -- 172.21.15.66:0/3926974555 learned_addr learned my addr 172.21.15.66:0/3926974555 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:04:40.252 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.251+0000 7ff1927fc640 10 client.?.objecter ms_handle_connect 0x7ff19c13f710 2024-03-25T18:04:40.252 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.251+0000 7ff1927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:40.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.254+0000 7ff1927fc640 10 client.?.objecter ms_handle_connect 0x7ff19c1430c0 2024-03-25T18:04:40.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.254+0000 7ff1927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:40.259 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.258+0000 7ff1927fc640 1 -- 172.21.15.66:0/3926974555 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 348492762 0 0) 0x7ff19c0b7520 con 0x7ff19c13f710 2024-03-25T18:04:40.259 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.258+0000 7ff1927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:04:40.259 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.258+0000 7ff1927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:40.259 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.258+0000 7ff1927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:40.259 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.258+0000 7ff1927fc640 10 monclient(hunting): my global_id is 4258 2024-03-25T18:04:40.260 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.258+0000 7ff1927fc640 1 -- 172.21.15.66:0/3926974555 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff180004eb0 con 0x7ff19c13f710 2024-03-25T18:04:40.268 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.267+0000 7ff1927fc640 1 -- 172.21.15.66:0/3926974555 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 631648862 0 0) 0x7ff19c135810 con 0x7ff19c1430c0 2024-03-25T18:04:40.268 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.267+0000 7ff1927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:04:40.268 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.267+0000 7ff1927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:40.268 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.267+0000 7ff1927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:40.268 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.267+0000 7ff1927fc640 10 monclient(hunting): my global_id is 4259 2024-03-25T18:04:40.268 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.267+0000 7ff1927fc640 1 -- 172.21.15.66:0/3926974555 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff19c0b7520 con 0x7ff19c1430c0 2024-03-25T18:04:40.269 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.268+0000 7ff1927fc640 10 client.?.objecter ms_handle_connect 0x7ff19c0a98f0 2024-03-25T18:04:40.269 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.268+0000 7ff1927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:40.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.273+0000 7ff1927fc640 1 -- 172.21.15.66:0/3926974555 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2220531500 0 0) 0x7ff180004eb0 con 0x7ff19c13f710 2024-03-25T18:04:40.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.273+0000 7ff1927fc640 1 -- 172.21.15.66:0/3926974555 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff19c135810 con 0x7ff19c13f710 2024-03-25T18:04:40.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.273+0000 7ff1927fc640 1 -- 172.21.15.66:0/3926974555 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7ff18c001a10 con 0x7ff19c13f710 2024-03-25T18:04:40.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.273+0000 7ff1927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:40.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.273+0000 7ff1927fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:04:40.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.273+0000 7ff1927fc640 10 monclient(hunting): dump: 2024-03-25T18:04:40.275 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:40.275 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:40.275 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:40.275 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:40.275 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:40.275 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:40.275 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:40.275 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:40.275 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:40.275 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:40.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.273+0000 7ff1927fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:04:40.285 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.284+0000 7ff1927fc640 1 -- 172.21.15.66:0/3926974555 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161493689 0 0) 0x7ff19c136ad0 con 0x7ff19c0a98f0 2024-03-25T18:04:40.285 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.284+0000 7ff1927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:04:40.285 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.284+0000 7ff1927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:40.285 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.284+0000 7ff1927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:40.285 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.284+0000 7ff1927fc640 10 monclient(hunting): my global_id is 4260 2024-03-25T18:04:40.285 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.284+0000 7ff1927fc640 1 -- 172.21.15.66:0/3926974555 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff180004eb0 con 0x7ff19c0a98f0 2024-03-25T18:04:40.300 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.299+0000 7ff1927fc640 1 -- 172.21.15.66:0/3926974555 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 819008754 0 0) 0x7ff180004eb0 con 0x7ff19c0a98f0 2024-03-25T18:04:40.300 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.299+0000 7ff1927fc640 1 -- 172.21.15.66:0/3926974555 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff19c136ad0 con 0x7ff19c0a98f0 2024-03-25T18:04:40.301 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.299+0000 7ff1927fc640 1 -- 172.21.15.66:0/3926974555 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7ff198001930 con 0x7ff19c0a98f0 2024-03-25T18:04:40.301 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.299+0000 7ff1927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:40.301 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.299+0000 7ff1927fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:04:40.301 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.299+0000 7ff1927fc640 10 monclient(hunting): dump: 2024-03-25T18:04:40.301 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:40.301 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:40.301 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:40.301 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:40.301 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:40.301 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:40.301 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:40.301 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:40.301 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:40.301 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:40.303 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.302+0000 7ff1927fc640 1 -- 172.21.15.66:0/3926974555 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 401207626 0 0) 0x7ff19c0b7520 con 0x7ff19c1430c0 2024-03-25T18:04:40.303 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.302+0000 7ff1927fc640 1 -- 172.21.15.66:0/3926974555 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff180004eb0 con 0x7ff19c1430c0 2024-03-25T18:04:40.303 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.302+0000 7ff1927fc640 1 -- 172.21.15.66:0/3926974555 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7ff194001e80 con 0x7ff19c1430c0 2024-03-25T18:04:40.303 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.302+0000 7ff1927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:40.303 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.302+0000 7ff1927fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:04:40.304 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.302+0000 7ff1927fc640 10 monclient(hunting): dump: 2024-03-25T18:04:40.304 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:40.304 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:40.304 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:40.304 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:40.304 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:40.304 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:40.304 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:40.304 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:40.304 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:40.304 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:40.305 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.303+0000 7ff1927fc640 1 -- 172.21.15.66:0/3926974555 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2229577475 0 0) 0x7ff19c135810 con 0x7ff19c13f710 2024-03-25T18:04:40.305 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.303+0000 7ff1927fc640 1 -- 172.21.15.66:0/3926974555 >> v1:172.21.15.66:6790/0 conn(0x7ff19c1430c0 legacy=0x7ff19c130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:40.305 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.303+0000 7ff1927fc640 1 -- 172.21.15.66:0/3926974555 >> v1:172.21.15.66:6789/0 conn(0x7ff19c0a98f0 legacy=0x7ff19c135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:40.305 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.303+0000 7ff1927fc640 10 monclient: _finish_hunting 0 2024-03-25T18:04:40.306 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.303+0000 7ff1927fc640 1 monclient: found mon.b 2024-03-25T18:04:40.306 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.303+0000 7ff1927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:40.306 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.303+0000 7ff1927fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:04:40.306 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.303+0000 7ff1927fc640 1 -- 172.21.15.66:0/3926974555 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff19c0b6180 con 0x7ff19c13f710 2024-03-25T18:04:40.306 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.303+0000 7ff1927fc640 10 monclient: _finish_auth 0 2024-03-25T18:04:40.306 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.303+0000 7ff1927fc640 10 monclient: _check_auth_tickets 2024-03-25T18:04:40.306 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.303+0000 7ff1927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:40.306 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.304+0000 7ff1a3e2a640 5 monclient: authenticate success, global_id 4258 2024-03-25T18:04:40.306 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.304+0000 7ff1a3e2a640 10 monclient: _renew_subs 2024-03-25T18:04:40.306 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.304+0000 7ff1a3e2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:04:40.306 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.304+0000 7ff1a3e2a640 1 -- 172.21.15.66:0/3926974555 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff19c0b7a40 con 0x7ff19c13f710 2024-03-25T18:04:40.306 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.304+0000 7ff1a3e2a640 10 client.4258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:04:40.306 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.304+0000 7ff1a3e2a640 10 monclient: _renew_subs 2024-03-25T18:04:40.307 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.304+0000 7ff1a3e2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:04:40.307 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.304+0000 7ff1a3e2a640 1 -- 172.21.15.66:0/3926974555 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff19c131840 con 0x7ff19c13f710 2024-03-25T18:04:40.307 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.304+0000 7ff1a3e2a640 1 librados: init done 2024-03-25T18:04:40.307 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.305+0000 7ff1a3e2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:40.307 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.305+0000 7ff1a3e2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:40.307 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.305+0000 7ff1a3e2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:04:40.308 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.305+0000 7ff1a3e2a640 1 -- 172.21.15.66:0/3926974555 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff19c136f60 con 0x7ff19c13f710 2024-03-25T18:04:40.587 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.585+0000 7ff1927fc640 1 -- 172.21.15.66:0/3926974555 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff18c001c70 con 0x7ff19c13f710 2024-03-25T18:04:40.587 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.585+0000 7ff1927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:40.587 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.585+0000 7ff1927fc640 1 -- 172.21.15.66:0/3926974555 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7ff18c0038c0 con 0x7ff19c13f710 2024-03-25T18:04:40.587 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.585+0000 7ff1927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:40.587 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.585+0000 7ff1927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:04:40.587 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.585+0000 7ff1927fc640 10 monclient: dump: 2024-03-25T18:04:40.587 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:40.587 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:40.587 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:40.587 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:40.587 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:40.587 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:40.588 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:40.588 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:40.588 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:40.588 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:40.588 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.585+0000 7ff193fff640 4 set_mon_vals no callback set 2024-03-25T18:04:40.588 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.586+0000 7ff1927fc640 1 -- 172.21.15.66:0/3926974555 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7ff18c003aa0 con 0x7ff19c13f710 2024-03-25T18:04:40.588 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.587+0000 7ff1a1bc6640 10 monclient: get_auth_request con 0x7ff18006e180 auth_method 0 2024-03-25T18:04:40.589 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.587+0000 7ff1927fc640 10 client.4258.objecter ms_handle_connect 0x7ff18006e180 2024-03-25T18:04:40.594 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.592+0000 7ff1927fc640 1 -- 172.21.15.66:0/3926974555 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7ff18c054740 con 0x7ff19c13f710 2024-03-25T18:04:40.594 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.592+0000 7ff1927fc640 10 client.4258.objecter ms_dispatch 0x7ff19c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:04:40.594 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.592+0000 7ff1927fc640 3 client.4258.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:04:40.594 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.592+0000 7ff1927fc640 3 client.4258.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:04:40.594 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.592+0000 7ff1927fc640 20 client.4258.objecter dump_active .. 0 homeless 2024-03-25T18:04:40.594 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.593+0000 7ff1927fc640 1 -- 172.21.15.66:0/3926974555 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff18c050080 con 0x7ff19c13f710 2024-03-25T18:04:40.594 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.593+0000 7ff1927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:40.595 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.593+0000 7ff1927fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:04:40.833 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.832+0000 7ff1a3e2a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:40.833 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.832+0000 7ff1a3e2a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:40.833 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.832+0000 7ff1a3e2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:04:40.834 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.832+0000 7ff1a3e2a640 1 -- 172.21.15.66:0/3926974555 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff19c0b76b0 con 0x7ff19c13f710 2024-03-25T18:04:40.913 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.912+0000 7ff1927fc640 1 -- 172.21.15.66:0/3926974555 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7ff18c052180 con 0x7ff19c13f710 2024-03-25T18:04:40.913 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.912+0000 7ff1927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:40.913 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.912+0000 7ff1927fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:04:40.914 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:04:40.914 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:04:40.916 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.914+0000 7ff1a3e2a640 10 librados: watch_flush enter 2024-03-25T18:04:40.916 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.914+0000 7ff1a3e2a640 10 librados: watch_flush exit 2024-03-25T18:04:40.916 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.914+0000 7ff1a3e2a640 20 client.4258.objecter shutdown clearing up homeless session... 2024-03-25T18:04:40.916 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.914+0000 7ff1a3e2a640 10 client.4258.objecter successfully canceled tick 2024-03-25T18:04:40.916 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.914+0000 7ff1a3e2a640 1 -- 172.21.15.66:0/3926974555 >> v1:172.21.15.193:6800/2275676803 conn(0x7ff18006e180 legacy=0x7ff180070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:40.917 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.914+0000 7ff1a3e2a640 10 monclient: shutdown 2024-03-25T18:04:40.917 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.914+0000 7ff1a3e2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:40.917 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.914+0000 7ff1a3e2a640 1 -- 172.21.15.66:0/3926974555 >> v1:172.21.15.193:6789/0 conn(0x7ff19c13f710 legacy=0x7ff19c0b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:40.917 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.914+0000 7ff1a3e2a640 1 -- 172.21.15.66:0/3926974555 shutdown_connections 2024-03-25T18:04:40.917 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.914+0000 7ff1a3e2a640 1 -- 172.21.15.66:0/3926974555 >> 172.21.15.66:0/3926974555 conn(0x7ff19c0b7ea0 msgr2=0x7ff19c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:40.917 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.914+0000 7ff1a3e2a640 1 -- 172.21.15.66:0/3926974555 shutdown_connections 2024-03-25T18:04:40.917 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.914+0000 7ff1a3e2a640 1 -- 172.21.15.66:0/3926974555 wait complete. 2024-03-25T18:04:40.917 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:40.914+0000 7ff1a3e2a640 1 librados: shutdown 2024-03-25T18:04:40.928 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:04:43.654 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:01:14.634 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:43.727 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:01:14.686 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:43.729 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:01:14.710 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:43.775 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:01:14.756 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:43.785 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:01:14.744 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:43.848 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:01:14.828 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:43.982 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:01:14.963 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:44.009 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:01:14.968 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:44.090 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:01:15.070 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:04:47.734 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:04:47.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.856+0000 7ff89c2c1640 10 monclient: get_monmap_and_config 2024-03-25T18:04:47.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.856+0000 7ff89c2c1640 10 monclient: build_initial_monmap 2024-03-25T18:04:47.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.856+0000 7ff89c2c1640 10 monclient: monmap: 2024-03-25T18:04:47.858 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:04:47.858 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:47.858 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:04:47.857712+0000 2024-03-25T18:04:47.858 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:04:47.857712+0000 2024-03-25T18:04:47.858 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:47.858 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:47.858 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:04:47.858 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:04:47.858 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:04:47.858 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:47.859 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.856+0000 7ff89c2c1640 1 Processor -- start 2024-03-25T18:04:47.860 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.857+0000 7ff89c2c1640 1 -- start start 2024-03-25T18:04:47.860 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.857+0000 7ff89c2c1640 10 monclient: init 2024-03-25T18:04:47.860 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.857+0000 7ff89c2c1640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:47.860 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.857+0000 7ff89c2c1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:04:47.860 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.857+0000 7ff89c2c1640 10 monclient(hunting): picked mon.noname-c con 0x7ff8940a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:04:47.860 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.857+0000 7ff89c2c1640 10 monclient(hunting): picked mon.noname-a con 0x7ff89413f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:04:47.860 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.857+0000 7ff89c2c1640 10 monclient(hunting): picked mon.noname-b con 0x7ff8941430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:04:47.860 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.857+0000 7ff89c2c1640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff894142520 con 0x7ff89413f710 2024-03-25T18:04:47.860 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.857+0000 7ff89c2c1640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8940b28a0 con 0x7ff8940a98f0 2024-03-25T18:04:47.860 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.857+0000 7ff89c2c1640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8940b2a30 con 0x7ff8941430c0 2024-03-25T18:04:47.860 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.857+0000 7ff89c2c1640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:47.860 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.857+0000 7ff89c2c1640 10 monclient(hunting): authenticate will time out at 7036.917966s 2024-03-25T18:04:47.866 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.865+0000 7ff89a05d640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7ff8940a98f0 0x7ff8940a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:51862/0 (socket says 172.21.15.66:51862) 2024-03-25T18:04:47.866 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.865+0000 7ff89a05d640 1 -- 172.21.15.66:0/3229137943 learned_addr learned my addr 172.21.15.66:0/3229137943 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:04:47.878 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.877+0000 7ff89905b640 1 -- 172.21.15.66:0/3229137943 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674292057 0 0) 0x7ff8940b28a0 con 0x7ff8940a98f0 2024-03-25T18:04:47.878 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.877+0000 7ff89905b640 10 monclient(hunting): _negotiate 2024-03-25T18:04:47.878 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.877+0000 7ff89905b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:47.879 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.877+0000 7ff89905b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:47.879 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.877+0000 7ff89905b640 10 monclient(hunting): my global_id is 4261 2024-03-25T18:04:47.879 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.877+0000 7ff89905b640 1 -- 172.21.15.66:0/3229137943 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff880003520 con 0x7ff8940a98f0 2024-03-25T18:04:47.890 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.889+0000 7ff89905b640 1 -- 172.21.15.66:0/3229137943 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1868899136 0 0) 0x7ff894142520 con 0x7ff89413f710 2024-03-25T18:04:47.890 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.889+0000 7ff89905b640 10 monclient(hunting): _negotiate 2024-03-25T18:04:47.890 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.889+0000 7ff89905b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:47.890 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.889+0000 7ff89905b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:47.890 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.889+0000 7ff89905b640 10 monclient(hunting): my global_id is 4263 2024-03-25T18:04:47.891 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.889+0000 7ff89905b640 1 -- 172.21.15.66:0/3229137943 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8940b28a0 con 0x7ff89413f710 2024-03-25T18:04:47.891 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.890+0000 7ff89905b640 1 -- 172.21.15.66:0/3229137943 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299872211 0 0) 0x7ff8940b2a30 con 0x7ff8941430c0 2024-03-25T18:04:47.891 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.890+0000 7ff89905b640 10 monclient(hunting): _negotiate 2024-03-25T18:04:47.891 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.890+0000 7ff89905b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:47.891 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.890+0000 7ff89905b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:47.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.890+0000 7ff89905b640 10 monclient(hunting): my global_id is 4262 2024-03-25T18:04:47.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.890+0000 7ff89905b640 1 -- 172.21.15.66:0/3229137943 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff894142520 con 0x7ff8941430c0 2024-03-25T18:04:47.902 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.900+0000 7ff89905b640 1 -- 172.21.15.66:0/3229137943 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2030380266 0 0) 0x7ff880003520 con 0x7ff8940a98f0 2024-03-25T18:04:47.902 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.900+0000 7ff89905b640 1 -- 172.21.15.66:0/3229137943 >> v1:172.21.15.66:6790/0 conn(0x7ff8941430c0 legacy=0x7ff89414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:47.902 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.900+0000 7ff89905b640 1 -- 172.21.15.66:0/3229137943 >> v1:172.21.15.66:6789/0 conn(0x7ff89413f710 legacy=0x7ff8941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:47.902 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.900+0000 7ff89905b640 10 monclient: _finish_hunting 0 2024-03-25T18:04:47.902 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.900+0000 7ff89905b640 1 monclient: found mon.noname-c 2024-03-25T18:04:47.902 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.900+0000 7ff89905b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:47.902 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.900+0000 7ff89905b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:04:47.902 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.900+0000 7ff89905b640 1 -- 172.21.15.66:0/3229137943 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8940b44c0 con 0x7ff8940a98f0 2024-03-25T18:04:47.902 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.900+0000 7ff89905b640 10 monclient: _finish_auth 0 2024-03-25T18:04:47.902 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.900+0000 7ff89905b640 10 monclient: _check_auth_tickets 2024-03-25T18:04:47.902 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.900+0000 7ff89905b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:47.902 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.900+0000 7ff89905b640 1 -- 172.21.15.66:0/3229137943 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7ff890001690 con 0x7ff8940a98f0 2024-03-25T18:04:47.902 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.900+0000 7ff89905b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:47.903 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.900+0000 7ff89905b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:04:47.903 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.900+0000 7ff89905b640 10 monclient: dump: 2024-03-25T18:04:47.903 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:47.903 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:47.903 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:47.903 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:47.903 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:47.903 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:47.903 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:47.903 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:47.903 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:47.903 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:47.903 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.900+0000 7ff89c2c1640 5 monclient: authenticate success, global_id 4261 2024-03-25T18:04:47.903 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.900+0000 7ff89c2c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:04:47.916 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.915+0000 7ff89905b640 1 -- 172.21.15.66:0/3229137943 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8900017f0 con 0x7ff8940a98f0 2024-03-25T18:04:47.916 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.915+0000 7ff89905b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:47.917 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.915+0000 7ff89905b640 1 -- 172.21.15.66:0/3229137943 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7ff890001c00 con 0x7ff8940a98f0 2024-03-25T18:04:47.917 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.915+0000 7ff89905b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:47.917 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.915+0000 7ff89905b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:04:47.917 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.915+0000 7ff89905b640 10 monclient: dump: 2024-03-25T18:04:47.917 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:47.917 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:47.917 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:47.917 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:47.917 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:47.917 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:47.917 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:47.917 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:47.917 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:47.917 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:47.917 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.915+0000 7ff89c2c1640 10 monclient: get_monmap_and_config success 2024-03-25T18:04:47.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.915+0000 7ff89c2c1640 4 set_mon_vals no callback set 2024-03-25T18:04:47.919 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.916+0000 7ff89c2c1640 10 monclient: shutdown 2024-03-25T18:04:47.919 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.916+0000 7ff89c2c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:47.919 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.916+0000 7ff89c2c1640 1 -- 172.21.15.66:0/3229137943 >> v1:172.21.15.193:6789/0 conn(0x7ff8940a98f0 legacy=0x7ff8940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:47.919 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.916+0000 7ff89c2c1640 1 -- 172.21.15.66:0/3229137943 shutdown_connections 2024-03-25T18:04:47.919 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.916+0000 7ff89c2c1640 1 -- 172.21.15.66:0/3229137943 >> 172.21.15.66:0/3229137943 conn(0x7ff8940b7ea0 msgr2=0x7ff8940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:47.919 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.916+0000 7ff89c2c1640 1 -- 172.21.15.66:0/3229137943 shutdown_connections 2024-03-25T18:04:47.919 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.916+0000 7ff89c2c1640 1 -- 172.21.15.66:0/3229137943 wait complete. 2024-03-25T18:04:47.920 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.917+0000 7ff89c2c1640 10 monclient: build_initial_monmap 2024-03-25T18:04:47.920 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.917+0000 7ff89c2c1640 10 monclient: monmap: 2024-03-25T18:04:47.920 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:04:47.920 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:04:47.920 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:04:47.920 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:04:47.920 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:47.920 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:04:47.920 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:04:47.920 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:04:47.920 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:04:47.920 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:47.920 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.917+0000 7ff89c2c1640 1 librados: starting msgr at 2024-03-25T18:04:47.920 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.917+0000 7ff89c2c1640 1 librados: starting objecter 2024-03-25T18:04:47.920 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.917+0000 7ff89c2c1640 1 Processor -- start 2024-03-25T18:04:47.920 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.918+0000 7ff89c2c1640 1 -- start start 2024-03-25T18:04:47.920 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.918+0000 7ff89c2c1640 1 librados: setting wanted keys 2024-03-25T18:04:47.921 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.918+0000 7ff89c2c1640 1 librados: calling monclient init 2024-03-25T18:04:47.921 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.918+0000 7ff89c2c1640 10 monclient: init 2024-03-25T18:04:47.922 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.918+0000 7ff89c2c1640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:47.922 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.918+0000 7ff89c2c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:04:47.922 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.918+0000 7ff89c2c1640 10 monclient(hunting): picked mon.noname-b con 0x7ff8940a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:04:47.922 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.918+0000 7ff89c2c1640 10 monclient(hunting): picked mon.noname-c con 0x7ff89413f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:04:47.922 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.919+0000 7ff89c2c1640 10 monclient(hunting): picked mon.noname-a con 0x7ff8941430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:04:47.922 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.919+0000 7ff89c2c1640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff894135590 con 0x7ff8941430c0 2024-03-25T18:04:47.922 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.919+0000 7ff89c2c1640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff894136e50 con 0x7ff8940a98f0 2024-03-25T18:04:47.922 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.919+0000 7ff89c2c1640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff894135070 con 0x7ff89413f710 2024-03-25T18:04:47.922 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.919+0000 7ff89c2c1640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:47.922 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.919+0000 7ff89c2c1640 10 monclient(hunting): authenticate will time out at 7036.979194s 2024-03-25T18:04:47.925 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.924+0000 7ff89985c640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7ff89413f710 0x7ff8941d7750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:47726/0 (socket says 172.21.15.66:47726) 2024-03-25T18:04:47.925 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.924+0000 7ff89985c640 1 -- 172.21.15.66:0/106064835 learned_addr learned my addr 172.21.15.66:0/106064835 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:04:47.928 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.927+0000 7ff88affd640 10 client.?.objecter ms_handle_connect 0x7ff89413f710 2024-03-25T18:04:47.929 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.927+0000 7ff88affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:47.933 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.931+0000 7ff88affd640 10 client.?.objecter ms_handle_connect 0x7ff8940a98f0 2024-03-25T18:04:47.933 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.931+0000 7ff88affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:47.940 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.939+0000 7ff88affd640 10 client.?.objecter ms_handle_connect 0x7ff8941430c0 2024-03-25T18:04:47.940 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.939+0000 7ff88affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:47.945 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.944+0000 7ff88affd640 1 -- 172.21.15.66:0/106064835 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2182189256 0 0) 0x7ff894136e50 con 0x7ff8940a98f0 2024-03-25T18:04:47.945 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.944+0000 7ff88affd640 10 monclient(hunting): _negotiate 2024-03-25T18:04:47.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.944+0000 7ff88affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:47.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.944+0000 7ff88affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:47.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.944+0000 7ff88affd640 10 monclient(hunting): my global_id is 4264 2024-03-25T18:04:47.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.944+0000 7ff88affd640 1 -- 172.21.15.66:0/106064835 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff878004eb0 con 0x7ff8940a98f0 2024-03-25T18:04:47.950 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.949+0000 7ff88affd640 1 -- 172.21.15.66:0/106064835 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 377362598 0 0) 0x7ff894135070 con 0x7ff89413f710 2024-03-25T18:04:47.951 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.949+0000 7ff88affd640 10 monclient(hunting): _negotiate 2024-03-25T18:04:47.951 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.949+0000 7ff88affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:47.951 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.949+0000 7ff88affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:47.951 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.949+0000 7ff88affd640 10 monclient(hunting): my global_id is 4265 2024-03-25T18:04:47.951 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.949+0000 7ff88affd640 1 -- 172.21.15.66:0/106064835 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff894136e50 con 0x7ff89413f710 2024-03-25T18:04:47.953 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.952+0000 7ff88affd640 1 -- 172.21.15.66:0/106064835 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 504997698 0 0) 0x7ff894135590 con 0x7ff8941430c0 2024-03-25T18:04:47.953 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.952+0000 7ff88affd640 10 monclient(hunting): _negotiate 2024-03-25T18:04:47.953 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.952+0000 7ff88affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:47.953 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.952+0000 7ff88affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:47.953 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.952+0000 7ff88affd640 10 monclient(hunting): my global_id is 4266 2024-03-25T18:04:47.953 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.952+0000 7ff88affd640 1 -- 172.21.15.66:0/106064835 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff894135070 con 0x7ff8941430c0 2024-03-25T18:04:47.958 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.957+0000 7ff88affd640 1 -- 172.21.15.66:0/106064835 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1084263336 0 0) 0x7ff878004eb0 con 0x7ff8940a98f0 2024-03-25T18:04:47.958 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.957+0000 7ff88affd640 1 -- 172.21.15.66:0/106064835 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff894135590 con 0x7ff8940a98f0 2024-03-25T18:04:47.958 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.957+0000 7ff88affd640 1 -- 172.21.15.66:0/106064835 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7ff890001b00 con 0x7ff8940a98f0 2024-03-25T18:04:47.958 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.957+0000 7ff88affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:47.958 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.957+0000 7ff88affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:04:47.958 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.957+0000 7ff88affd640 10 monclient(hunting): dump: 2024-03-25T18:04:47.958 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:47.958 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:47.959 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:47.959 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:47.959 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:47.959 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:47.959 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:47.959 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:47.959 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:47.959 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:47.959 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.957+0000 7ff88affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:04:47.966 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.965+0000 7ff88affd640 1 -- 172.21.15.66:0/106064835 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 547148569 0 0) 0x7ff894136e50 con 0x7ff89413f710 2024-03-25T18:04:47.966 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.965+0000 7ff88affd640 1 -- 172.21.15.66:0/106064835 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff878004eb0 con 0x7ff89413f710 2024-03-25T18:04:47.966 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.965+0000 7ff88affd640 1 -- 172.21.15.66:0/106064835 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7ff8840019b0 con 0x7ff89413f710 2024-03-25T18:04:47.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.965+0000 7ff88affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:47.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.965+0000 7ff88affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:04:47.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.965+0000 7ff88affd640 10 monclient(hunting): dump: 2024-03-25T18:04:47.967 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:47.967 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:47.967 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:47.967 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:47.967 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:47.967 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:47.967 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:47.967 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:47.967 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:47.967 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:47.976 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.974+0000 7ff88affd640 1 -- 172.21.15.66:0/106064835 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1199516395 0 0) 0x7ff894135590 con 0x7ff8940a98f0 2024-03-25T18:04:47.976 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.974+0000 7ff88affd640 1 -- 172.21.15.66:0/106064835 >> v1:172.21.15.66:6790/0 conn(0x7ff89413f710 legacy=0x7ff8941d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:47.976 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.974+0000 7ff88affd640 1 -- 172.21.15.66:0/106064835 >> v1:172.21.15.66:6789/0 conn(0x7ff8941430c0 legacy=0x7ff8941e3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:47.976 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.974+0000 7ff88affd640 10 monclient: _finish_hunting 0 2024-03-25T18:04:47.976 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.974+0000 7ff88affd640 1 monclient: found mon.b 2024-03-25T18:04:47.976 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.974+0000 7ff88affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:47.976 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.974+0000 7ff88affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:04:47.976 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.974+0000 7ff88affd640 1 -- 172.21.15.66:0/106064835 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff894135d80 con 0x7ff8940a98f0 2024-03-25T18:04:47.977 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.974+0000 7ff88affd640 10 monclient: _finish_auth 0 2024-03-25T18:04:47.977 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.974+0000 7ff88affd640 10 monclient: _check_auth_tickets 2024-03-25T18:04:47.977 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.974+0000 7ff88affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:47.977 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.974+0000 7ff89c2c1640 5 monclient: authenticate success, global_id 4264 2024-03-25T18:04:47.977 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.974+0000 7ff89c2c1640 10 monclient: _renew_subs 2024-03-25T18:04:47.977 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.974+0000 7ff89c2c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:04:47.977 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.974+0000 7ff89c2c1640 1 -- 172.21.15.66:0/106064835 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff894131660 con 0x7ff8940a98f0 2024-03-25T18:04:47.977 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.974+0000 7ff89c2c1640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:04:47.977 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.974+0000 7ff89c2c1640 10 monclient: _renew_subs 2024-03-25T18:04:47.977 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.974+0000 7ff89c2c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:04:47.977 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.974+0000 7ff89c2c1640 1 -- 172.21.15.66:0/106064835 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8940b78a0 con 0x7ff8940a98f0 2024-03-25T18:04:47.977 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.974+0000 7ff89c2c1640 1 librados: init done 2024-03-25T18:04:47.978 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.975+0000 7ff89c2c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:47.978 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.975+0000 7ff89c2c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:47.978 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.975+0000 7ff89c2c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:04:47.978 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:47.975+0000 7ff89c2c1640 1 -- 172.21.15.66:0/106064835 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff894135f70 con 0x7ff8940a98f0 2024-03-25T18:04:48.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.251+0000 7ff88affd640 1 -- 172.21.15.66:0/106064835 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff890001220 con 0x7ff8940a98f0 2024-03-25T18:04:48.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.251+0000 7ff88affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:48.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.251+0000 7ff88affd640 1 -- 172.21.15.66:0/106064835 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7ff890004a90 con 0x7ff8940a98f0 2024-03-25T18:04:48.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.251+0000 7ff88affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:48.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.251+0000 7ff88affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:04:48.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.251+0000 7ff88affd640 10 monclient: dump: 2024-03-25T18:04:48.253 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:48.253 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:48.253 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:48.253 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:48.253 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:48.253 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:48.254 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:48.254 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:48.254 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:48.254 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:48.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.251+0000 7ff89885a640 4 set_mon_vals no callback set 2024-03-25T18:04:48.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.252+0000 7ff88affd640 1 -- 172.21.15.66:0/106064835 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7ff89001aa60 con 0x7ff8940a98f0 2024-03-25T18:04:48.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.253+0000 7ff89985c640 10 monclient: get_auth_request con 0x7ff87806df60 auth_method 0 2024-03-25T18:04:48.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.253+0000 7ff88affd640 10 client.4264.objecter ms_handle_connect 0x7ff87806df60 2024-03-25T18:04:48.258 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.257+0000 7ff88affd640 1 -- 172.21.15.66:0/106064835 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7ff890054670 con 0x7ff8940a98f0 2024-03-25T18:04:48.258 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.257+0000 7ff88affd640 10 client.4264.objecter ms_dispatch 0x7ff894000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:04:48.258 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.257+0000 7ff88affd640 3 client.4264.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:04:48.258 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.257+0000 7ff88affd640 3 client.4264.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:04:48.258 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.257+0000 7ff88affd640 20 client.4264.objecter dump_active .. 0 homeless 2024-03-25T18:04:48.259 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.257+0000 7ff88affd640 1 -- 172.21.15.66:0/106064835 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff8900b18b0 con 0x7ff8940a98f0 2024-03-25T18:04:48.259 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.257+0000 7ff88affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:48.259 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.257+0000 7ff88affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:04:48.505 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.503+0000 7ff89c2c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:48.505 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.503+0000 7ff89c2c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:48.505 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.503+0000 7ff89c2c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:04:48.505 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.503+0000 7ff89c2c1640 1 -- 172.21.15.66:0/106064835 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff894135200 con 0x7ff8940a98f0 2024-03-25T18:04:48.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.618+0000 7ff88affd640 1 -- 172.21.15.66:0/106064835 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7ff89004ffb0 con 0x7ff8940a98f0 2024-03-25T18:04:48.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.618+0000 7ff88affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:48.620 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.618+0000 7ff88affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:04:48.620 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:04:48.620 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:04:48.622 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.620+0000 7ff89c2c1640 10 librados: watch_flush enter 2024-03-25T18:04:48.622 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.620+0000 7ff89c2c1640 10 librados: watch_flush exit 2024-03-25T18:04:48.622 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.620+0000 7ff89c2c1640 20 client.4264.objecter shutdown clearing up homeless session... 2024-03-25T18:04:48.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.620+0000 7ff89c2c1640 10 client.4264.objecter successfully canceled tick 2024-03-25T18:04:48.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.620+0000 7ff89c2c1640 1 -- 172.21.15.66:0/106064835 >> v1:172.21.15.193:6800/2275676803 conn(0x7ff87806df60 legacy=0x7ff878070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:48.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.620+0000 7ff89c2c1640 10 monclient: shutdown 2024-03-25T18:04:48.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.620+0000 7ff89c2c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:48.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.620+0000 7ff89c2c1640 1 -- 172.21.15.66:0/106064835 >> v1:172.21.15.193:6789/0 conn(0x7ff8940a98f0 legacy=0x7ff894130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:48.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.620+0000 7ff89c2c1640 1 -- 172.21.15.66:0/106064835 shutdown_connections 2024-03-25T18:04:48.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.620+0000 7ff89c2c1640 1 -- 172.21.15.66:0/106064835 >> 172.21.15.66:0/106064835 conn(0x7ff8940b7ea0 msgr2=0x7ff894074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:48.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.621+0000 7ff89c2c1640 1 -- 172.21.15.66:0/106064835 shutdown_connections 2024-03-25T18:04:48.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.621+0000 7ff89c2c1640 1 -- 172.21.15.66:0/106064835 wait complete. 2024-03-25T18:04:48.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:48.621+0000 7ff89c2c1640 1 librados: shutdown 2024-03-25T18:04:48.635 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:04:55.439 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:04:55.557 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.556+0000 7f1f476fb640 10 monclient: get_monmap_and_config 2024-03-25T18:04:55.557 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.556+0000 7f1f476fb640 10 monclient: build_initial_monmap 2024-03-25T18:04:55.557 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.556+0000 7f1f476fb640 10 monclient: monmap: 2024-03-25T18:04:55.557 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:04:55.557 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:55.557 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:04:55.557802+0000 2024-03-25T18:04:55.557 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:04:55.557802+0000 2024-03-25T18:04:55.557 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:55.558 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:55.558 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:04:55.558 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:04:55.558 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:04:55.558 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:55.559 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.556+0000 7f1f476fb640 1 Processor -- start 2024-03-25T18:04:55.560 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.557+0000 7f1f476fb640 1 -- start start 2024-03-25T18:04:55.560 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.557+0000 7f1f476fb640 10 monclient: init 2024-03-25T18:04:55.560 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.557+0000 7f1f476fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:55.560 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.557+0000 7f1f476fb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:04:55.560 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.557+0000 7f1f476fb640 10 monclient(hunting): picked mon.noname-b con 0x7f1f400a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:04:55.560 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.557+0000 7f1f476fb640 10 monclient(hunting): picked mon.noname-c con 0x7f1f4013f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:04:55.560 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.557+0000 7f1f476fb640 10 monclient(hunting): picked mon.noname-a con 0x7f1f401430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:04:55.560 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.557+0000 7f1f476fb640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f40142520 con 0x7f1f401430c0 2024-03-25T18:04:55.560 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.557+0000 7f1f476fb640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f400b28a0 con 0x7f1f4013f710 2024-03-25T18:04:55.560 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.557+0000 7f1f476fb640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f400b2a30 con 0x7f1f400a98f0 2024-03-25T18:04:55.560 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.557+0000 7f1f476fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:55.560 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.558+0000 7f1f476fb640 10 monclient(hunting): authenticate will time out at 7044.618090s 2024-03-25T18:04:55.560 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.558+0000 7f1f45497640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f1f400a98f0 0x7f1f400a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:46138/0 (socket says 172.21.15.66:46138) 2024-03-25T18:04:55.560 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.558+0000 7f1f45497640 1 -- 172.21.15.66:0/772068597 learned_addr learned my addr 172.21.15.66:0/772068597 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:04:55.634 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.633+0000 7f1f37fff640 1 -- 172.21.15.66:0/772068597 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2736792818 0 0) 0x7f1f40142520 con 0x7f1f401430c0 2024-03-25T18:04:55.634 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.633+0000 7f1f37fff640 10 monclient(hunting): _negotiate 2024-03-25T18:04:55.634 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.633+0000 7f1f37fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:55.634 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.633+0000 7f1f37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:55.634 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.633+0000 7f1f37fff640 10 monclient(hunting): my global_id is 4269 2024-03-25T18:04:55.634 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.633+0000 7f1f37fff640 1 -- 172.21.15.66:0/772068597 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f2c003520 con 0x7f1f401430c0 2024-03-25T18:04:55.640 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.639+0000 7f1f37fff640 1 -- 172.21.15.66:0/772068597 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1257811463 0 0) 0x7f1f400b28a0 con 0x7f1f4013f710 2024-03-25T18:04:55.640 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.639+0000 7f1f37fff640 10 monclient(hunting): _negotiate 2024-03-25T18:04:55.640 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.639+0000 7f1f37fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:55.640 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.639+0000 7f1f37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:55.640 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.639+0000 7f1f37fff640 10 monclient(hunting): my global_id is 4267 2024-03-25T18:04:55.640 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.639+0000 7f1f37fff640 1 -- 172.21.15.66:0/772068597 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f40142520 con 0x7f1f4013f710 2024-03-25T18:04:55.652 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.651+0000 7f1f37fff640 1 -- 172.21.15.66:0/772068597 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1922299450 0 0) 0x7f1f40142520 con 0x7f1f4013f710 2024-03-25T18:04:55.652 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.651+0000 7f1f37fff640 1 -- 172.21.15.66:0/772068597 >> v1:172.21.15.66:6790/0 conn(0x7f1f400a98f0 legacy=0x7f1f400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:55.652 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.651+0000 7f1f37fff640 1 -- 172.21.15.66:0/772068597 >> v1:172.21.15.66:6789/0 conn(0x7f1f401430c0 legacy=0x7f1f4014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:55.652 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.651+0000 7f1f37fff640 10 monclient: _finish_hunting 0 2024-03-25T18:04:55.652 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.651+0000 7f1f37fff640 1 monclient: found mon.noname-c 2024-03-25T18:04:55.652 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.651+0000 7f1f37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:55.652 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.651+0000 7f1f37fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:04:55.652 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.651+0000 7f1f37fff640 1 -- 172.21.15.66:0/772068597 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f400b44c0 con 0x7f1f4013f710 2024-03-25T18:04:55.652 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.651+0000 7f1f37fff640 10 monclient: _finish_auth 0 2024-03-25T18:04:55.653 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.651+0000 7f1f37fff640 10 monclient: _check_auth_tickets 2024-03-25T18:04:55.653 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.651+0000 7f1f37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:55.653 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.651+0000 7f1f37fff640 1 -- 172.21.15.66:0/772068597 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f1f30001670 con 0x7f1f4013f710 2024-03-25T18:04:55.653 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.651+0000 7f1f37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:55.653 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.651+0000 7f1f37fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:04:55.653 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.651+0000 7f1f37fff640 10 monclient: dump: 2024-03-25T18:04:55.653 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:55.653 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:55.653 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:55.653 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:55.653 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:55.653 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:55.653 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:55.653 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:55.653 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:55.654 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:55.654 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.651+0000 7f1f476fb640 5 monclient: authenticate success, global_id 4267 2024-03-25T18:04:55.654 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.651+0000 7f1f476fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:04:55.693 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.692+0000 7f1f37fff640 1 -- 172.21.15.66:0/772068597 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f300017d0 con 0x7f1f4013f710 2024-03-25T18:04:55.693 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.692+0000 7f1f37fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:55.693 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.692+0000 7f1f37fff640 1 -- 172.21.15.66:0/772068597 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f1f30001ba0 con 0x7f1f4013f710 2024-03-25T18:04:55.693 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.692+0000 7f1f476fb640 10 monclient: get_monmap_and_config success 2024-03-25T18:04:55.694 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.692+0000 7f1f476fb640 4 set_mon_vals no callback set 2024-03-25T18:04:55.694 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.692+0000 7f1f37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:55.694 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.692+0000 7f1f37fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:04:55.694 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.692+0000 7f1f37fff640 10 monclient: dump: 2024-03-25T18:04:55.694 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:55.694 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:55.694 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:55.694 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:55.694 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:55.694 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:55.694 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:55.694 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:55.694 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:55.694 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:55.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.693+0000 7f1f476fb640 10 monclient: shutdown 2024-03-25T18:04:55.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.693+0000 7f1f476fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:55.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.693+0000 7f1f476fb640 1 -- 172.21.15.66:0/772068597 >> v1:172.21.15.193:6789/0 conn(0x7f1f4013f710 legacy=0x7f1f401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:55.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.693+0000 7f1f476fb640 1 -- 172.21.15.66:0/772068597 shutdown_connections 2024-03-25T18:04:55.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.693+0000 7f1f476fb640 1 -- 172.21.15.66:0/772068597 >> 172.21.15.66:0/772068597 conn(0x7f1f400b7ea0 msgr2=0x7f1f400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:55.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.693+0000 7f1f476fb640 1 -- 172.21.15.66:0/772068597 shutdown_connections 2024-03-25T18:04:55.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.693+0000 7f1f476fb640 1 -- 172.21.15.66:0/772068597 wait complete. 2024-03-25T18:04:55.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.694+0000 7f1f476fb640 10 monclient: build_initial_monmap 2024-03-25T18:04:55.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.694+0000 7f1f476fb640 10 monclient: monmap: 2024-03-25T18:04:55.696 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:04:55.696 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:04:55.697 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:04:55.697 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:04:55.697 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:04:55.697 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:04:55.697 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:04:55.697 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:04:55.697 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:04:55.697 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:55.697 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.695+0000 7f1f476fb640 1 librados: starting msgr at 2024-03-25T18:04:55.697 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.695+0000 7f1f476fb640 1 librados: starting objecter 2024-03-25T18:04:55.697 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.695+0000 7f1f476fb640 1 Processor -- start 2024-03-25T18:04:55.697 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.695+0000 7f1f476fb640 1 -- start start 2024-03-25T18:04:55.697 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.695+0000 7f1f476fb640 1 librados: setting wanted keys 2024-03-25T18:04:55.697 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.695+0000 7f1f476fb640 1 librados: calling monclient init 2024-03-25T18:04:55.697 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.695+0000 7f1f476fb640 10 monclient: init 2024-03-25T18:04:55.698 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.696+0000 7f1f476fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:04:55.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.696+0000 7f1f476fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:04:55.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.696+0000 7f1f476fb640 10 monclient(hunting): picked mon.noname-b con 0x7f1f400a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:04:55.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.696+0000 7f1f476fb640 10 monclient(hunting): picked mon.noname-c con 0x7f1f4013f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:04:55.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.696+0000 7f1f476fb640 10 monclient(hunting): picked mon.noname-a con 0x7f1f401430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:04:55.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.696+0000 7f1f476fb640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f401388b0 con 0x7f1f401430c0 2024-03-25T18:04:55.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.696+0000 7f1f476fb640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f401397c0 con 0x7f1f400a98f0 2024-03-25T18:04:55.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.696+0000 7f1f476fb640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f40138150 con 0x7f1f4013f710 2024-03-25T18:04:55.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.696+0000 7f1f476fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:04:55.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.696+0000 7f1f476fb640 10 monclient(hunting): authenticate will time out at 7044.756442s 2024-03-25T18:04:55.717 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.716+0000 7f1f45c98640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f1f401430c0 0x7f1f401e1ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:43704/0 (socket says 172.21.15.66:43704) 2024-03-25T18:04:55.717 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.716+0000 7f1f45c98640 1 -- 172.21.15.66:0/3115665764 learned_addr learned my addr 172.21.15.66:0/3115665764 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:04:55.717 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.716+0000 7f1f35ffb640 10 client.?.objecter ms_handle_connect 0x7f1f401430c0 2024-03-25T18:04:55.717 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.716+0000 7f1f35ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:55.725 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.724+0000 7f1f35ffb640 10 client.?.objecter ms_handle_connect 0x7f1f400a98f0 2024-03-25T18:04:55.725 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.724+0000 7f1f35ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:55.731 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.730+0000 7f1f35ffb640 10 client.?.objecter ms_handle_connect 0x7f1f4013f710 2024-03-25T18:04:55.731 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.730+0000 7f1f35ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:04:55.733 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.732+0000 7f1f35ffb640 1 -- 172.21.15.66:0/3115665764 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3757762330 0 0) 0x7f1f401388b0 con 0x7f1f401430c0 2024-03-25T18:04:55.733 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.732+0000 7f1f35ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:04:55.733 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.732+0000 7f1f35ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:55.733 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.732+0000 7f1f35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:55.733 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.732+0000 7f1f35ffb640 10 monclient(hunting): my global_id is 4272 2024-03-25T18:04:55.733 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.732+0000 7f1f35ffb640 1 -- 172.21.15.66:0/3115665764 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f24004eb0 con 0x7f1f401430c0 2024-03-25T18:04:55.735 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.734+0000 7f1f35ffb640 1 -- 172.21.15.66:0/3115665764 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937279269 0 0) 0x7f1f401397c0 con 0x7f1f400a98f0 2024-03-25T18:04:55.735 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.734+0000 7f1f35ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:04:55.735 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.734+0000 7f1f35ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:55.735 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.734+0000 7f1f35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:55.735 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.734+0000 7f1f35ffb640 10 monclient(hunting): my global_id is 4270 2024-03-25T18:04:55.735 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.734+0000 7f1f35ffb640 1 -- 172.21.15.66:0/3115665764 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f401388b0 con 0x7f1f400a98f0 2024-03-25T18:04:55.744 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.743+0000 7f1f35ffb640 1 -- 172.21.15.66:0/3115665764 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3134857293 0 0) 0x7f1f40138150 con 0x7f1f4013f710 2024-03-25T18:04:55.744 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.743+0000 7f1f35ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:04:55.745 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.743+0000 7f1f35ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:04:55.745 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.743+0000 7f1f35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:04:55.745 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.743+0000 7f1f35ffb640 10 monclient(hunting): my global_id is 4271 2024-03-25T18:04:55.745 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.743+0000 7f1f35ffb640 1 -- 172.21.15.66:0/3115665764 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f401397c0 con 0x7f1f4013f710 2024-03-25T18:04:55.746 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.744+0000 7f1f35ffb640 1 -- 172.21.15.66:0/3115665764 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3114634804 0 0) 0x7f1f401388b0 con 0x7f1f400a98f0 2024-03-25T18:04:55.746 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.744+0000 7f1f35ffb640 1 -- 172.21.15.66:0/3115665764 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f40138150 con 0x7f1f400a98f0 2024-03-25T18:04:55.746 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.744+0000 7f1f35ffb640 1 -- 172.21.15.66:0/3115665764 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f1f3c0018a0 con 0x7f1f400a98f0 2024-03-25T18:04:55.746 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.744+0000 7f1f35ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:55.746 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.744+0000 7f1f35ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:04:55.746 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.744+0000 7f1f35ffb640 10 monclient(hunting): dump: 2024-03-25T18:04:55.746 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:55.746 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:55.746 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:55.746 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:55.746 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:55.746 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:55.746 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:55.747 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:55.747 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:55.747 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:55.747 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.744+0000 7f1f35ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:04:55.759 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.758+0000 7f1f35ffb640 1 -- 172.21.15.66:0/3115665764 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 855486024 0 0) 0x7f1f401397c0 con 0x7f1f4013f710 2024-03-25T18:04:55.759 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.758+0000 7f1f35ffb640 1 -- 172.21.15.66:0/3115665764 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f401388b0 con 0x7f1f4013f710 2024-03-25T18:04:55.759 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.758+0000 7f1f35ffb640 1 -- 172.21.15.66:0/3115665764 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f1f30001530 con 0x7f1f4013f710 2024-03-25T18:04:55.759 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.758+0000 7f1f35ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:55.760 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.758+0000 7f1f35ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:04:55.760 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.758+0000 7f1f35ffb640 10 monclient(hunting): dump: 2024-03-25T18:04:55.760 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:55.760 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:55.760 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:55.760 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:55.760 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:55.760 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:55.760 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:55.760 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:55.760 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:55.760 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:55.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.760+0000 7f1f35ffb640 1 -- 172.21.15.66:0/3115665764 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2355996597 0 0) 0x7f1f40138150 con 0x7f1f400a98f0 2024-03-25T18:04:55.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.760+0000 7f1f35ffb640 1 -- 172.21.15.66:0/3115665764 >> v1:172.21.15.66:6790/0 conn(0x7f1f4013f710 legacy=0x7f1f401da8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:55.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.760+0000 7f1f35ffb640 1 -- 172.21.15.66:0/3115665764 >> v1:172.21.15.66:6789/0 conn(0x7f1f401430c0 legacy=0x7f1f401e1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:55.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.760+0000 7f1f35ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:04:55.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.760+0000 7f1f35ffb640 1 monclient: found mon.b 2024-03-25T18:04:55.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.760+0000 7f1f35ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:04:55.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.760+0000 7f1f35ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:04:55.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.760+0000 7f1f35ffb640 1 -- 172.21.15.66:0/3115665764 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f401382e0 con 0x7f1f400a98f0 2024-03-25T18:04:55.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.760+0000 7f1f35ffb640 10 monclient: _finish_auth 0 2024-03-25T18:04:55.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.760+0000 7f1f35ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:04:55.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.760+0000 7f1f35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:04:55.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.760+0000 7f1f476fb640 5 monclient: authenticate success, global_id 4270 2024-03-25T18:04:55.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.760+0000 7f1f476fb640 10 monclient: _renew_subs 2024-03-25T18:04:55.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.760+0000 7f1f476fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:04:55.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.760+0000 7f1f476fb640 1 -- 172.21.15.66:0/3115665764 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f40138530 con 0x7f1f400a98f0 2024-03-25T18:04:55.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.761+0000 7f1f476fb640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:04:55.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.761+0000 7f1f476fb640 10 monclient: _renew_subs 2024-03-25T18:04:55.764 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.761+0000 7f1f476fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:04:55.764 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.761+0000 7f1f476fb640 1 -- 172.21.15.66:0/3115665764 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f401db1e0 con 0x7f1f400a98f0 2024-03-25T18:04:55.764 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.761+0000 7f1f476fb640 1 librados: init done 2024-03-25T18:04:55.765 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.762+0000 7f1f476fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:55.765 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.762+0000 7f1f476fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:55.765 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.762+0000 7f1f476fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:04:55.765 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:55.762+0000 7f1f476fb640 1 -- 172.21.15.66:0/3115665764 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f4014f2d0 con 0x7f1f400a98f0 2024-03-25T18:04:56.038 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.037+0000 7f1f35ffb640 1 -- 172.21.15.66:0/3115665764 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f3c001cb0 con 0x7f1f400a98f0 2024-03-25T18:04:56.038 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.037+0000 7f1f35ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:04:56.038 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.037+0000 7f1f35ffb640 1 -- 172.21.15.66:0/3115665764 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f1f3c004980 con 0x7f1f400a98f0 2024-03-25T18:04:56.038 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.037+0000 7f1f35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:04:56.038 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.037+0000 7f1f35ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:04:56.038 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.037+0000 7f1f35ffb640 10 monclient: dump: 2024-03-25T18:04:56.039 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:04:56.039 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:04:56.039 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:56.039 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:04:56.039 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:04:56.039 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:04:56.039 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:04:56.039 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:04:56.039 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:04:56.039 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:04:56.039 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.037+0000 7f1f377fe640 4 set_mon_vals no callback set 2024-03-25T18:04:56.040 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.038+0000 7f1f35ffb640 1 -- 172.21.15.66:0/3115665764 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f1f3c004ae0 con 0x7f1f400a98f0 2024-03-25T18:04:56.040 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.039+0000 7f1f44c96640 10 monclient: get_auth_request con 0x7f1f2406df30 auth_method 0 2024-03-25T18:04:56.040 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.039+0000 7f1f35ffb640 10 client.4270.objecter ms_handle_connect 0x7f1f2406df30 2024-03-25T18:04:56.042 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.041+0000 7f1f35ffb640 1 -- 172.21.15.66:0/3115665764 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f1f3c054680 con 0x7f1f400a98f0 2024-03-25T18:04:56.042 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.041+0000 7f1f35ffb640 10 client.4270.objecter ms_dispatch 0x7f1f40000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:04:56.042 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.041+0000 7f1f35ffb640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:04:56.042 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.041+0000 7f1f35ffb640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:04:56.042 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.041+0000 7f1f35ffb640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-25T18:04:56.044 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.043+0000 7f1f35ffb640 1 -- 172.21.15.66:0/3115665764 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1f3c04ffc0 con 0x7f1f400a98f0 2024-03-25T18:04:56.044 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.043+0000 7f1f35ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:04:56.044 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.043+0000 7f1f35ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:04:56.300 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.299+0000 7f1f476fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:56.300 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.299+0000 7f1f476fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:56.300 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.299+0000 7f1f476fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:04:56.300 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.299+0000 7f1f476fb640 1 -- 172.21.15.66:0/3115665764 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f400627b0 con 0x7f1f400a98f0 2024-03-25T18:04:56.331 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.329+0000 7f1f35ffb640 1 -- 172.21.15.66:0/3115665764 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f1f3c0520c0 con 0x7f1f400a98f0 2024-03-25T18:04:56.331 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.330+0000 7f1f35ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:04:56.331 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.330+0000 7f1f35ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:04:56.331 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:04:56.331 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:04:56.334 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.332+0000 7f1f476fb640 10 librados: watch_flush enter 2024-03-25T18:04:56.334 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.332+0000 7f1f476fb640 10 librados: watch_flush exit 2024-03-25T18:04:56.334 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.332+0000 7f1f476fb640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-25T18:04:56.334 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.332+0000 7f1f476fb640 10 client.4270.objecter successfully canceled tick 2024-03-25T18:04:56.334 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.332+0000 7f1f476fb640 1 -- 172.21.15.66:0/3115665764 >> v1:172.21.15.193:6800/2275676803 conn(0x7f1f2406df30 legacy=0x7f1f240703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:56.334 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.332+0000 7f1f476fb640 10 monclient: shutdown 2024-03-25T18:04:56.334 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.332+0000 7f1f476fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:04:56.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.332+0000 7f1f476fb640 1 -- 172.21.15.66:0/3115665764 >> v1:172.21.15.193:6789/0 conn(0x7f1f400a98f0 legacy=0x7f1f40137990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:04:56.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.332+0000 7f1f476fb640 1 -- 172.21.15.66:0/3115665764 shutdown_connections 2024-03-25T18:04:56.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.332+0000 7f1f476fb640 1 -- 172.21.15.66:0/3115665764 >> 172.21.15.66:0/3115665764 conn(0x7f1f400b7ea0 msgr2=0x7f1f40074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:04:56.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.332+0000 7f1f476fb640 1 -- 172.21.15.66:0/3115665764 shutdown_connections 2024-03-25T18:04:56.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.332+0000 7f1f476fb640 1 -- 172.21.15.66:0/3115665764 wait complete. 2024-03-25T18:04:56.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:04:56.333+0000 7f1f476fb640 1 librados: shutdown 2024-03-25T18:04:56.345 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:05:03.152 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:05:03.271 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.270+0000 7fb887f32640 10 monclient: get_monmap_and_config 2024-03-25T18:05:03.271 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.270+0000 7fb887f32640 10 monclient: build_initial_monmap 2024-03-25T18:05:03.271 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.270+0000 7fb887f32640 10 monclient: monmap: 2024-03-25T18:05:03.271 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:05:03.271 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:03.271 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:05:03.271740+0000 2024-03-25T18:05:03.271 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:05:03.271740+0000 2024-03-25T18:05:03.272 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:03.272 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:03.272 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:05:03.272 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:05:03.272 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:05:03.272 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:03.273 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.270+0000 7fb887f32640 1 Processor -- start 2024-03-25T18:05:03.273 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.271+0000 7fb887f32640 1 -- start start 2024-03-25T18:05:03.273 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.271+0000 7fb887f32640 10 monclient: init 2024-03-25T18:05:03.273 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.272+0000 7fb887f32640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:03.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.272+0000 7fb887f32640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:05:03.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.272+0000 7fb887f32640 10 monclient(hunting): picked mon.noname-b con 0x7fb8800705b0 addr v1:172.21.15.66:6790/0 2024-03-25T18:05:03.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.272+0000 7fb887f32640 10 monclient(hunting): picked mon.noname-a con 0x7fb8801402d0 addr v1:172.21.15.66:6789/0 2024-03-25T18:05:03.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.272+0000 7fb887f32640 10 monclient(hunting): picked mon.noname-c con 0x7fb880149ae0 addr v1:172.21.15.193:6789/0 2024-03-25T18:05:03.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.272+0000 7fb887f32640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8800718c0 con 0x7fb8801402d0 2024-03-25T18:05:03.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.272+0000 7fb887f32640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8800714c0 con 0x7fb880149ae0 2024-03-25T18:05:03.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.272+0000 7fb887f32640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb880072b50 con 0x7fb8800705b0 2024-03-25T18:05:03.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.272+0000 7fb887f32640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:03.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.272+0000 7fb887f32640 10 monclient(hunting): authenticate will time out at 7052.332327s 2024-03-25T18:05:03.279 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.278+0000 7fb8854cd640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7fb8801402d0 0x7fb880140730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:48186/0 (socket says 172.21.15.66:48186) 2024-03-25T18:05:03.279 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.278+0000 7fb8854cd640 1 -- 172.21.15.66:0/3193747832 learned_addr learned my addr 172.21.15.66:0/3193747832 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:05:03.286 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.285+0000 7fb884ccc640 1 -- 172.21.15.66:0/3193747832 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543638834 0 0) 0x7fb8800718c0 con 0x7fb8801402d0 2024-03-25T18:05:03.286 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.285+0000 7fb884ccc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:03.287 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.285+0000 7fb884ccc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:03.287 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.285+0000 7fb884ccc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:03.287 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.285+0000 7fb884ccc640 10 monclient(hunting): my global_id is 4275 2024-03-25T18:05:03.287 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.285+0000 7fb884ccc640 1 -- 172.21.15.66:0/3193747832 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb86c003520 con 0x7fb8801402d0 2024-03-25T18:05:03.301 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.300+0000 7fb884ccc640 1 -- 172.21.15.66:0/3193747832 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2003905484 0 0) 0x7fb86c003520 con 0x7fb8801402d0 2024-03-25T18:05:03.301 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.300+0000 7fb884ccc640 1 -- 172.21.15.66:0/3193747832 >> v1:172.21.15.66:6790/0 conn(0x7fb8800705b0 legacy=0x7fb88013d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:03.302 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.300+0000 7fb884ccc640 1 -- 172.21.15.66:0/3193747832 >> v1:172.21.15.193:6789/0 conn(0x7fb880149ae0 legacy=0x7fb88014bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:03.302 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.300+0000 7fb884ccc640 10 monclient: _finish_hunting 0 2024-03-25T18:05:03.302 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.300+0000 7fb884ccc640 1 monclient: found mon.noname-a 2024-03-25T18:05:03.302 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.300+0000 7fb884ccc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:03.302 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.300+0000 7fb884ccc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.66:6789/0 2024-03-25T18:05:03.302 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.300+0000 7fb884ccc640 1 -- 172.21.15.66:0/3193747832 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb88013dd10 con 0x7fb8801402d0 2024-03-25T18:05:03.302 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.300+0000 7fb884ccc640 10 monclient: _finish_auth 0 2024-03-25T18:05:03.302 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.300+0000 7fb884ccc640 10 monclient: _check_auth_tickets 2024-03-25T18:05:03.302 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.300+0000 7fb884ccc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:03.302 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.300+0000 7fb884ccc640 1 -- 172.21.15.66:0/3193747832 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fb870001690 con 0x7fb8801402d0 2024-03-25T18:05:03.302 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.300+0000 7fb884ccc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:03.302 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.300+0000 7fb884ccc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:05:03.302 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.300+0000 7fb884ccc640 10 monclient: dump: 2024-03-25T18:05:03.303 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:03.303 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:03.303 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:03.303 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:03.303 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:03.303 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:03.303 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:03.303 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:03.303 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:03.303 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:03.303 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.300+0000 7fb887f32640 5 monclient: authenticate success, global_id 4275 2024-03-25T18:05:03.303 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.300+0000 7fb887f32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:03.305 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.304+0000 7fb884ccc640 1 -- 172.21.15.66:0/3193747832 <== mon.0 v1:172.21.15.66:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8700017f0 con 0x7fb8801402d0 2024-03-25T18:05:03.305 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.304+0000 7fb884ccc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:03.305 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.304+0000 7fb887f32640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:03.305 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.304+0000 7fb887f32640 4 set_mon_vals no callback set 2024-03-25T18:05:03.307 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.305+0000 7fb884ccc640 1 -- 172.21.15.66:0/3193747832 <== mon.0 v1:172.21.15.66:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fb870001c00 con 0x7fb8801402d0 2024-03-25T18:05:03.307 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.305+0000 7fb884ccc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:03.307 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.305+0000 7fb884ccc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:03.307 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.305+0000 7fb884ccc640 10 monclient: dump: 2024-03-25T18:05:03.307 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:03.307 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:03.307 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:03.307 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:03.307 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:03.307 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:03.307 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:03.307 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:03.307 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:03.308 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:03.308 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.305+0000 7fb887f32640 10 monclient: shutdown 2024-03-25T18:05:03.308 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.305+0000 7fb887f32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:03.308 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.305+0000 7fb887f32640 1 -- 172.21.15.66:0/3193747832 >> v1:172.21.15.66:6789/0 conn(0x7fb8801402d0 legacy=0x7fb880140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:03.308 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.305+0000 7fb887f32640 1 -- 172.21.15.66:0/3193747832 shutdown_connections 2024-03-25T18:05:03.308 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.305+0000 7fb887f32640 1 -- 172.21.15.66:0/3193747832 >> 172.21.15.66:0/3193747832 conn(0x7fb8800b5d10 msgr2=0x7fb8800b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:03.308 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.305+0000 7fb887f32640 1 -- 172.21.15.66:0/3193747832 shutdown_connections 2024-03-25T18:05:03.308 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.305+0000 7fb887f32640 1 -- 172.21.15.66:0/3193747832 wait complete. 2024-03-25T18:05:03.310 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.306+0000 7fb887f32640 10 monclient: build_initial_monmap 2024-03-25T18:05:03.310 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.306+0000 7fb887f32640 10 monclient: monmap: 2024-03-25T18:05:03.310 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:05:03.310 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:03.310 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:05:03.310 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:05:03.310 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:03.311 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:05:03.311 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:05:03.311 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:05:03.311 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:05:03.311 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:03.311 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.307+0000 7fb887f32640 1 librados: starting msgr at 2024-03-25T18:05:03.311 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.307+0000 7fb887f32640 1 librados: starting objecter 2024-03-25T18:05:03.311 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.307+0000 7fb887f32640 1 Processor -- start 2024-03-25T18:05:03.311 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.307+0000 7fb887f32640 1 -- start start 2024-03-25T18:05:03.311 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.307+0000 7fb887f32640 1 librados: setting wanted keys 2024-03-25T18:05:03.311 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.307+0000 7fb887f32640 1 librados: calling monclient init 2024-03-25T18:05:03.312 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.307+0000 7fb887f32640 10 monclient: init 2024-03-25T18:05:03.312 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.308+0000 7fb887f32640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:03.312 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.308+0000 7fb887f32640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:05:03.312 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.308+0000 7fb887f32640 10 monclient(hunting): picked mon.noname-c con 0x7fb8800705b0 addr v1:172.21.15.66:6790/0 2024-03-25T18:05:03.312 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.308+0000 7fb887f32640 10 monclient(hunting): picked mon.noname-a con 0x7fb8801402d0 addr v1:172.21.15.66:6789/0 2024-03-25T18:05:03.312 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.308+0000 7fb887f32640 10 monclient(hunting): picked mon.noname-b con 0x7fb880149ae0 addr v1:172.21.15.193:6789/0 2024-03-25T18:05:03.312 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.308+0000 7fb887f32640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb880141f90 con 0x7fb8801402d0 2024-03-25T18:05:03.312 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.308+0000 7fb887f32640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb880142270 con 0x7fb880149ae0 2024-03-25T18:05:03.312 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.308+0000 7fb887f32640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb88013fe40 con 0x7fb8800705b0 2024-03-25T18:05:03.312 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.308+0000 7fb887f32640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:03.312 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.308+0000 7fb887f32640 10 monclient(hunting): authenticate will time out at 7052.368934s 2024-03-25T18:05:03.322 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.321+0000 7fb8864cf640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fb880149ae0 0x7fb8801d5e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:57438/0 (socket says 172.21.15.66:57438) 2024-03-25T18:05:03.322 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.321+0000 7fb8864cf640 1 -- 172.21.15.66:0/1257625622 learned_addr learned my addr 172.21.15.66:0/1257625622 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:05:03.322 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.321+0000 7fb8767fc640 10 client.?.objecter ms_handle_connect 0x7fb880149ae0 2024-03-25T18:05:03.322 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.321+0000 7fb8767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:03.324 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.323+0000 7fb8767fc640 10 client.?.objecter ms_handle_connect 0x7fb8801402d0 2024-03-25T18:05:03.324 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.323+0000 7fb8767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:03.332 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.331+0000 7fb8767fc640 10 client.?.objecter ms_handle_connect 0x7fb8800705b0 2024-03-25T18:05:03.332 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.331+0000 7fb8767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:03.334 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.333+0000 7fb8767fc640 1 -- 172.21.15.66:0/1257625622 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1779282497 0 0) 0x7fb880142270 con 0x7fb880149ae0 2024-03-25T18:05:03.334 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.333+0000 7fb8767fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:03.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.333+0000 7fb8767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:03.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.333+0000 7fb8767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:03.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.333+0000 7fb8767fc640 10 monclient(hunting): my global_id is 4276 2024-03-25T18:05:03.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.333+0000 7fb8767fc640 1 -- 172.21.15.66:0/1257625622 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb864004eb0 con 0x7fb880149ae0 2024-03-25T18:05:03.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.336+0000 7fb8767fc640 1 -- 172.21.15.66:0/1257625622 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1028804385 0 0) 0x7fb880141f90 con 0x7fb8801402d0 2024-03-25T18:05:03.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.336+0000 7fb8767fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:03.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.336+0000 7fb8767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:03.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.336+0000 7fb8767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:03.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.336+0000 7fb8767fc640 10 monclient(hunting): my global_id is 4278 2024-03-25T18:05:03.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.336+0000 7fb8767fc640 1 -- 172.21.15.66:0/1257625622 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb880142270 con 0x7fb8801402d0 2024-03-25T18:05:03.348 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.347+0000 7fb8767fc640 1 -- 172.21.15.66:0/1257625622 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3416033093 0 0) 0x7fb864004eb0 con 0x7fb880149ae0 2024-03-25T18:05:03.348 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.347+0000 7fb8767fc640 1 -- 172.21.15.66:0/1257625622 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb880141f90 con 0x7fb880149ae0 2024-03-25T18:05:03.348 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.347+0000 7fb8767fc640 1 -- 172.21.15.66:0/1257625622 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fb878001c30 con 0x7fb880149ae0 2024-03-25T18:05:03.348 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.347+0000 7fb8767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:03.348 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.347+0000 7fb8767fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:05:03.348 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.347+0000 7fb8767fc640 10 monclient(hunting): dump: 2024-03-25T18:05:03.349 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:03.349 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:03.349 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:03.349 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:03.349 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:03.349 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:03.349 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:03.349 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:03.349 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:03.349 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:03.349 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.347+0000 7fb8767fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:03.350 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.349+0000 7fb8767fc640 1 -- 172.21.15.66:0/1257625622 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113326935 0 0) 0x7fb88013fe40 con 0x7fb8800705b0 2024-03-25T18:05:03.350 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.349+0000 7fb8767fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:03.350 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.349+0000 7fb8767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:03.351 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.349+0000 7fb8767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:03.351 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.349+0000 7fb8767fc640 10 monclient(hunting): my global_id is 4277 2024-03-25T18:05:03.351 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.349+0000 7fb8767fc640 1 -- 172.21.15.66:0/1257625622 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb864004eb0 con 0x7fb8800705b0 2024-03-25T18:05:03.351 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.350+0000 7fb8767fc640 1 -- 172.21.15.66:0/1257625622 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 314873339 0 0) 0x7fb880142270 con 0x7fb8801402d0 2024-03-25T18:05:03.351 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.350+0000 7fb8767fc640 1 -- 172.21.15.66:0/1257625622 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb88013fe40 con 0x7fb8801402d0 2024-03-25T18:05:03.351 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.350+0000 7fb8767fc640 1 -- 172.21.15.66:0/1257625622 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fb870001b00 con 0x7fb8801402d0 2024-03-25T18:05:03.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.350+0000 7fb8767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:03.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.350+0000 7fb8767fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:03.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.350+0000 7fb8767fc640 10 monclient(hunting): dump: 2024-03-25T18:05:03.352 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:03.352 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:03.352 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:03.352 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:03.352 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:03.352 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:03.352 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:03.352 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:03.352 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:03.352 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:03.362 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.361+0000 7fb8767fc640 1 -- 172.21.15.66:0/1257625622 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3428402024 0 0) 0x7fb864004eb0 con 0x7fb8800705b0 2024-03-25T18:05:03.362 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.361+0000 7fb8767fc640 1 -- 172.21.15.66:0/1257625622 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb880142270 con 0x7fb8800705b0 2024-03-25T18:05:03.363 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.361+0000 7fb8767fc640 1 -- 172.21.15.66:0/1257625622 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fb87c001830 con 0x7fb8800705b0 2024-03-25T18:05:03.363 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.361+0000 7fb8767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:03.363 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.361+0000 7fb8767fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:05:03.363 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.361+0000 7fb8767fc640 10 monclient(hunting): dump: 2024-03-25T18:05:03.363 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:03.363 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:03.363 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:03.363 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:03.363 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:03.363 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:03.363 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:03.363 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:03.364 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:03.364 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:03.367 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.365+0000 7fb8767fc640 1 -- 172.21.15.66:0/1257625622 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3501744148 0 0) 0x7fb880141f90 con 0x7fb880149ae0 2024-03-25T18:05:03.367 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.365+0000 7fb8767fc640 1 -- 172.21.15.66:0/1257625622 >> v1:172.21.15.66:6790/0 conn(0x7fb8800705b0 legacy=0x7fb88013eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:03.367 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.365+0000 7fb8767fc640 1 -- 172.21.15.66:0/1257625622 >> v1:172.21.15.66:6789/0 conn(0x7fb8801402d0 legacy=0x7fb8801d5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:03.367 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.365+0000 7fb8767fc640 10 monclient: _finish_hunting 0 2024-03-25T18:05:03.367 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.365+0000 7fb8767fc640 1 monclient: found mon.b 2024-03-25T18:05:03.367 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.365+0000 7fb8767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:03.367 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.365+0000 7fb8767fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:05:03.367 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.365+0000 7fb8767fc640 1 -- 172.21.15.66:0/1257625622 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb88013f860 con 0x7fb880149ae0 2024-03-25T18:05:03.367 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.365+0000 7fb8767fc640 10 monclient: _finish_auth 0 2024-03-25T18:05:03.367 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.365+0000 7fb8767fc640 10 monclient: _check_auth_tickets 2024-03-25T18:05:03.367 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.365+0000 7fb8767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:03.368 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.365+0000 7fb887f32640 5 monclient: authenticate success, global_id 4276 2024-03-25T18:05:03.368 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.365+0000 7fb887f32640 10 monclient: _renew_subs 2024-03-25T18:05:03.368 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.365+0000 7fb887f32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:05:03.368 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.365+0000 7fb887f32640 1 -- 172.21.15.66:0/1257625622 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb88013fb30 con 0x7fb880149ae0 2024-03-25T18:05:03.368 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.365+0000 7fb887f32640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:03.368 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.365+0000 7fb887f32640 10 monclient: _renew_subs 2024-03-25T18:05:03.368 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.365+0000 7fb887f32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:05:03.368 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.365+0000 7fb887f32640 1 -- 172.21.15.66:0/1257625622 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8801e1240 con 0x7fb880149ae0 2024-03-25T18:05:03.368 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.365+0000 7fb887f32640 1 librados: init done 2024-03-25T18:05:03.369 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.366+0000 7fb887f32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:03.369 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.366+0000 7fb887f32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:03.369 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.366+0000 7fb887f32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:05:03.369 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.366+0000 7fb887f32640 1 -- 172.21.15.66:0/1257625622 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb880141270 con 0x7fb880149ae0 2024-03-25T18:05:03.647 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.646+0000 7fb8767fc640 1 -- 172.21.15.66:0/1257625622 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb878003990 con 0x7fb880149ae0 2024-03-25T18:05:03.647 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.646+0000 7fb8767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:03.648 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.646+0000 7fb8767fc640 1 -- 172.21.15.66:0/1257625622 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fb878003ce0 con 0x7fb880149ae0 2024-03-25T18:05:03.648 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.646+0000 7fb8767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:03.648 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.646+0000 7fb8767fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:03.648 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.646+0000 7fb8767fc640 10 monclient: dump: 2024-03-25T18:05:03.648 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:03.648 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:03.648 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:03.648 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:03.648 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:03.648 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:03.648 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:03.648 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:03.648 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:03.648 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:03.648 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.646+0000 7fb877fff640 4 set_mon_vals no callback set 2024-03-25T18:05:03.649 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.647+0000 7fb8767fc640 1 -- 172.21.15.66:0/1257625622 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7fb878003f00 con 0x7fb880149ae0 2024-03-25T18:05:03.649 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.648+0000 7fb885cce640 10 monclient: get_auth_request con 0x7fb86406e180 auth_method 0 2024-03-25T18:05:03.649 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.648+0000 7fb8767fc640 10 client.4276.objecter ms_handle_connect 0x7fb86406e180 2024-03-25T18:05:03.652 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.651+0000 7fb8767fc640 1 -- 172.21.15.66:0/1257625622 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7fb878054a60 con 0x7fb880149ae0 2024-03-25T18:05:03.653 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.651+0000 7fb8767fc640 10 client.4276.objecter ms_dispatch 0x7fb880000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:05:03.653 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.651+0000 7fb8767fc640 3 client.4276.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:05:03.653 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.651+0000 7fb8767fc640 3 client.4276.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:05:03.653 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.651+0000 7fb8767fc640 20 client.4276.objecter dump_active .. 0 homeless 2024-03-25T18:05:03.653 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.651+0000 7fb8767fc640 1 -- 172.21.15.66:0/1257625622 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb878050570 con 0x7fb880149ae0 2024-03-25T18:05:03.653 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.651+0000 7fb8767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:03.653 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.651+0000 7fb8767fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:05:03.698 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:01:34.679 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:03.758 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:01:34.739 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:03.810 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:01:34.791 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:03.835 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:01:34.794 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:03.841 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:01:34.822 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:03.847 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:01:34.806 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:03.849 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:01:34.830 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:03.869 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.867+0000 7fb887f32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:03.869 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.867+0000 7fb887f32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:03.869 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.867+0000 7fb887f32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:05:03.869 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.867+0000 7fb887f32640 1 -- 172.21.15.66:0/1257625622 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8800627b0 con 0x7fb880149ae0 2024-03-25T18:05:03.898 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:01:34.857 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:03.913 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:01:34.893 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:03.954 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.953+0000 7fb8767fc640 1 -- 172.21.15.66:0/1257625622 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7fb8780524a0 con 0x7fb880149ae0 2024-03-25T18:05:03.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.953+0000 7fb8767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:03.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.953+0000 7fb8767fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:05:03.955 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:05:03.955 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:05:03.957 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.955+0000 7fb887f32640 10 librados: watch_flush enter 2024-03-25T18:05:03.957 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.956+0000 7fb887f32640 10 librados: watch_flush exit 2024-03-25T18:05:03.957 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.956+0000 7fb887f32640 20 client.4276.objecter shutdown clearing up homeless session... 2024-03-25T18:05:03.957 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.956+0000 7fb887f32640 10 client.4276.objecter successfully canceled tick 2024-03-25T18:05:03.958 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.956+0000 7fb887f32640 1 -- 172.21.15.66:0/1257625622 >> v1:172.21.15.193:6800/2275676803 conn(0x7fb86406e180 legacy=0x7fb864070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:03.958 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.956+0000 7fb887f32640 10 monclient: shutdown 2024-03-25T18:05:03.958 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.956+0000 7fb887f32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:03.958 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.956+0000 7fb887f32640 1 -- 172.21.15.66:0/1257625622 >> v1:172.21.15.193:6789/0 conn(0x7fb880149ae0 legacy=0x7fb8801d5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:03.958 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.956+0000 7fb887f32640 1 -- 172.21.15.66:0/1257625622 shutdown_connections 2024-03-25T18:05:03.958 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.956+0000 7fb887f32640 1 -- 172.21.15.66:0/1257625622 >> 172.21.15.66:0/1257625622 conn(0x7fb8800b5d10 msgr2=0x7fb880140730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:03.958 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.956+0000 7fb887f32640 1 -- 172.21.15.66:0/1257625622 shutdown_connections 2024-03-25T18:05:03.958 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.956+0000 7fb887f32640 1 -- 172.21.15.66:0/1257625622 wait complete. 2024-03-25T18:05:03.958 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:03.956+0000 7fb887f32640 1 librados: shutdown 2024-03-25T18:05:03.968 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:05:10.773 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:05:10.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.892+0000 7f7500250640 10 monclient: get_monmap_and_config 2024-03-25T18:05:10.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.892+0000 7f7500250640 10 monclient: build_initial_monmap 2024-03-25T18:05:10.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.892+0000 7f7500250640 10 monclient: monmap: 2024-03-25T18:05:10.894 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:05:10.894 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:10.894 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:05:10.893982+0000 2024-03-25T18:05:10.894 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:05:10.893982+0000 2024-03-25T18:05:10.894 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:10.894 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:10.894 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:05:10.894 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:05:10.894 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:05:10.894 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:10.895 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.893+0000 7f7500250640 1 Processor -- start 2024-03-25T18:05:10.895 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.893+0000 7f7500250640 1 -- start start 2024-03-25T18:05:10.895 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.893+0000 7f7500250640 10 monclient: init 2024-03-25T18:05:10.896 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.894+0000 7f7500250640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:10.896 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.894+0000 7f7500250640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:05:10.896 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.894+0000 7f7500250640 10 monclient(hunting): picked mon.noname-a con 0x7f74f80a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:05:10.896 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.894+0000 7f7500250640 10 monclient(hunting): picked mon.noname-c con 0x7f74f813f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:05:10.896 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.894+0000 7f7500250640 10 monclient(hunting): picked mon.noname-b con 0x7f74f81430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:05:10.896 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.894+0000 7f7500250640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f8142520 con 0x7f74f80a98f0 2024-03-25T18:05:10.897 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.894+0000 7f7500250640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f80b28a0 con 0x7f74f813f710 2024-03-25T18:05:10.897 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.894+0000 7f7500250640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f80b2a30 con 0x7f74f81430c0 2024-03-25T18:05:10.897 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.894+0000 7f7500250640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:10.897 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.894+0000 7f7500250640 10 monclient(hunting): authenticate will time out at 7059.954816s 2024-03-25T18:05:10.902 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.901+0000 7f74fd7eb640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f74f813f710 0x7f74f81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:41312/0 (socket says 172.21.15.66:41312) 2024-03-25T18:05:10.902 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.901+0000 7f74fd7eb640 1 -- 172.21.15.66:0/1604944466 learned_addr learned my addr 172.21.15.66:0/1604944466 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:05:10.922 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.921+0000 7f74fcfea640 1 -- 172.21.15.66:0/1604944466 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2960390165 0 0) 0x7f74f80b28a0 con 0x7f74f813f710 2024-03-25T18:05:10.922 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.921+0000 7f74fcfea640 10 monclient(hunting): _negotiate 2024-03-25T18:05:10.922 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.921+0000 7f74fcfea640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:10.922 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.921+0000 7f74fcfea640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:10.922 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.921+0000 7f74fcfea640 10 monclient(hunting): my global_id is 4279 2024-03-25T18:05:10.922 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.921+0000 7f74fcfea640 1 -- 172.21.15.66:0/1604944466 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74e4003520 con 0x7f74f813f710 2024-03-25T18:05:10.931 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.930+0000 7f74fcfea640 1 -- 172.21.15.66:0/1604944466 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2986951491 0 0) 0x7f74f80b2a30 con 0x7f74f81430c0 2024-03-25T18:05:10.931 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.930+0000 7f74fcfea640 10 monclient(hunting): _negotiate 2024-03-25T18:05:10.931 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.930+0000 7f74fcfea640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:10.931 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.930+0000 7f74fcfea640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:10.932 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.930+0000 7f74fcfea640 10 monclient(hunting): my global_id is 4280 2024-03-25T18:05:10.932 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.930+0000 7f74fcfea640 1 -- 172.21.15.66:0/1604944466 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74f80b28a0 con 0x7f74f81430c0 2024-03-25T18:05:10.941 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.939+0000 7f74fcfea640 1 -- 172.21.15.66:0/1604944466 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2340444713 0 0) 0x7f74e4003520 con 0x7f74f813f710 2024-03-25T18:05:10.941 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.939+0000 7f74fcfea640 1 -- 172.21.15.66:0/1604944466 >> v1:172.21.15.66:6790/0 conn(0x7f74f81430c0 legacy=0x7f74f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:10.941 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.939+0000 7f74fcfea640 1 -- 172.21.15.66:0/1604944466 >> v1:172.21.15.66:6789/0 conn(0x7f74f80a98f0 legacy=0x7f74f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:10.941 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.939+0000 7f74fcfea640 10 monclient: _finish_hunting 0 2024-03-25T18:05:10.941 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.939+0000 7f74fcfea640 1 monclient: found mon.noname-c 2024-03-25T18:05:10.941 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.939+0000 7f74fcfea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:10.941 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.939+0000 7f74fcfea640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:05:10.942 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.939+0000 7f74fcfea640 1 -- 172.21.15.66:0/1604944466 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f80b44c0 con 0x7f74f813f710 2024-03-25T18:05:10.942 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.939+0000 7f74fcfea640 10 monclient: _finish_auth 0 2024-03-25T18:05:10.942 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.939+0000 7f74fcfea640 10 monclient: _check_auth_tickets 2024-03-25T18:05:10.942 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.939+0000 7f74fcfea640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:10.942 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.939+0000 7f74fcfea640 1 -- 172.21.15.66:0/1604944466 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f74e8001690 con 0x7f74f813f710 2024-03-25T18:05:10.942 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.939+0000 7f7500250640 5 monclient: authenticate success, global_id 4279 2024-03-25T18:05:10.942 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.939+0000 7f7500250640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:10.942 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.939+0000 7f74fcfea640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:10.942 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.939+0000 7f74fcfea640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:05:10.942 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.939+0000 7f74fcfea640 10 monclient: dump: 2024-03-25T18:05:10.942 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:10.942 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:10.942 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:10.943 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:10.943 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:10.943 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:10.943 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:10.943 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:10.943 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:10.943 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:10.943 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.939+0000 7f7500250640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:10.951 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.950+0000 7f74fcfea640 1 -- 172.21.15.66:0/1604944466 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74e80017f0 con 0x7f74f813f710 2024-03-25T18:05:10.951 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.950+0000 7f74fcfea640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:10.951 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.950+0000 7f74fcfea640 1 -- 172.21.15.66:0/1604944466 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f74e8001c00 con 0x7f74f813f710 2024-03-25T18:05:10.951 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.950+0000 7f7500250640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:10.951 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.950+0000 7f7500250640 4 set_mon_vals no callback set 2024-03-25T18:05:10.952 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.950+0000 7f74fcfea640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:10.952 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.950+0000 7f74fcfea640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:10.952 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.950+0000 7f74fcfea640 10 monclient: dump: 2024-03-25T18:05:10.952 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:10.952 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:10.952 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:10.952 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:10.952 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:10.952 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:10.952 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:10.952 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:10.952 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:10.952 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:10.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.951+0000 7f7500250640 10 monclient: shutdown 2024-03-25T18:05:10.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.951+0000 7f7500250640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:10.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.951+0000 7f7500250640 1 -- 172.21.15.66:0/1604944466 >> v1:172.21.15.193:6789/0 conn(0x7f74f813f710 legacy=0x7f74f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:10.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.951+0000 7f7500250640 1 -- 172.21.15.66:0/1604944466 shutdown_connections 2024-03-25T18:05:10.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.951+0000 7f7500250640 1 -- 172.21.15.66:0/1604944466 >> 172.21.15.66:0/1604944466 conn(0x7f74f80b7ea0 msgr2=0x7f74f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:10.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.951+0000 7f7500250640 1 -- 172.21.15.66:0/1604944466 shutdown_connections 2024-03-25T18:05:10.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.951+0000 7f7500250640 1 -- 172.21.15.66:0/1604944466 wait complete. 2024-03-25T18:05:10.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.952+0000 7f7500250640 10 monclient: build_initial_monmap 2024-03-25T18:05:10.956 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.952+0000 7f7500250640 10 monclient: monmap: 2024-03-25T18:05:10.956 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:05:10.956 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:10.956 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:05:10.956 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:05:10.956 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:10.956 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:05:10.956 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:05:10.956 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:05:10.956 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:05:10.956 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:10.956 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.952+0000 7f7500250640 1 librados: starting msgr at 2024-03-25T18:05:10.956 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.952+0000 7f7500250640 1 librados: starting objecter 2024-03-25T18:05:10.957 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.952+0000 7f7500250640 1 Processor -- start 2024-03-25T18:05:10.957 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.952+0000 7f7500250640 1 -- start start 2024-03-25T18:05:10.957 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.952+0000 7f7500250640 1 librados: setting wanted keys 2024-03-25T18:05:10.957 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.952+0000 7f7500250640 1 librados: calling monclient init 2024-03-25T18:05:10.957 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.952+0000 7f7500250640 10 monclient: init 2024-03-25T18:05:10.957 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.953+0000 7f7500250640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:10.957 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.953+0000 7f7500250640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:05:10.957 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.953+0000 7f7500250640 10 monclient(hunting): picked mon.noname-a con 0x7f74f80a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:05:10.957 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.953+0000 7f7500250640 10 monclient(hunting): picked mon.noname-c con 0x7f74f813f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:05:10.957 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.953+0000 7f7500250640 10 monclient(hunting): picked mon.noname-b con 0x7f74f81430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:05:10.957 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.953+0000 7f7500250640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f81dc070 con 0x7f74f80a98f0 2024-03-25T18:05:10.957 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.953+0000 7f7500250640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f8137750 con 0x7f74f81430c0 2024-03-25T18:05:10.957 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.953+0000 7f7500250640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f8138360 con 0x7f74f813f710 2024-03-25T18:05:10.957 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.953+0000 7f7500250640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:10.958 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.953+0000 7f7500250640 10 monclient(hunting): authenticate will time out at 7060.013445s 2024-03-25T18:05:10.958 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.956+0000 7f74fdfec640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f74f80a98f0 0x7f74f8137c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:42670/0 (socket says 172.21.15.66:42670) 2024-03-25T18:05:10.958 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.956+0000 7f74fdfec640 1 -- 172.21.15.66:0/150859846 learned_addr learned my addr 172.21.15.66:0/150859846 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:05:10.960 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.959+0000 7f74ee7fc640 10 client.?.objecter ms_handle_connect 0x7f74f80a98f0 2024-03-25T18:05:10.960 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.959+0000 7f74ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:10.965 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.964+0000 7f74ee7fc640 10 client.?.objecter ms_handle_connect 0x7f74f81430c0 2024-03-25T18:05:10.965 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.964+0000 7f74ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:10.971 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.970+0000 7f74ee7fc640 10 client.?.objecter ms_handle_connect 0x7f74f813f710 2024-03-25T18:05:10.971 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.970+0000 7f74ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:10.971 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.970+0000 7f74ee7fc640 1 -- 172.21.15.66:0/150859846 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1866498816 0 0) 0x7f74f8137750 con 0x7f74f81430c0 2024-03-25T18:05:10.972 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.970+0000 7f74ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:10.972 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.970+0000 7f74ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:10.972 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.970+0000 7f74ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:10.972 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.970+0000 7f74ee7fc640 10 monclient(hunting): my global_id is 4282 2024-03-25T18:05:10.972 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.970+0000 7f74ee7fc640 1 -- 172.21.15.66:0/150859846 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74dc004eb0 con 0x7f74f81430c0 2024-03-25T18:05:10.972 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.971+0000 7f74ee7fc640 1 -- 172.21.15.66:0/150859846 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3007287272 0 0) 0x7f74f81dc070 con 0x7f74f80a98f0 2024-03-25T18:05:10.972 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.971+0000 7f74ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:10.972 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.971+0000 7f74ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:10.972 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.971+0000 7f74ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:10.973 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.971+0000 7f74ee7fc640 10 monclient(hunting): my global_id is 4284 2024-03-25T18:05:10.973 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.971+0000 7f74ee7fc640 1 -- 172.21.15.66:0/150859846 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74f8137750 con 0x7f74f80a98f0 2024-03-25T18:05:10.977 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.976+0000 7f74ee7fc640 1 -- 172.21.15.66:0/150859846 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554727763 0 0) 0x7f74f8138360 con 0x7f74f813f710 2024-03-25T18:05:10.977 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.976+0000 7f74ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:10.977 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.976+0000 7f74ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:10.978 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.976+0000 7f74ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:10.978 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.976+0000 7f74ee7fc640 10 monclient(hunting): my global_id is 4283 2024-03-25T18:05:10.978 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.976+0000 7f74ee7fc640 1 -- 172.21.15.66:0/150859846 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74f81dc070 con 0x7f74f813f710 2024-03-25T18:05:10.984 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.983+0000 7f74ee7fc640 1 -- 172.21.15.66:0/150859846 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4169434111 0 0) 0x7f74dc004eb0 con 0x7f74f81430c0 2024-03-25T18:05:10.985 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.983+0000 7f74ee7fc640 1 -- 172.21.15.66:0/150859846 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74f8138360 con 0x7f74f81430c0 2024-03-25T18:05:10.985 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.983+0000 7f74ee7fc640 1 -- 172.21.15.66:0/150859846 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f74f0001bd0 con 0x7f74f81430c0 2024-03-25T18:05:10.985 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.983+0000 7f74ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:10.985 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.983+0000 7f74ee7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:05:10.985 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.983+0000 7f74ee7fc640 10 monclient(hunting): dump: 2024-03-25T18:05:10.985 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:10.985 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:10.985 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:10.985 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:10.985 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:10.985 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:10.985 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:10.985 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:10.985 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:10.985 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:10.986 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.983+0000 7f74ee7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:10.989 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.988+0000 7f74ee7fc640 1 -- 172.21.15.66:0/150859846 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 985344359 0 0) 0x7f74f8137750 con 0x7f74f80a98f0 2024-03-25T18:05:10.989 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.988+0000 7f74ee7fc640 1 -- 172.21.15.66:0/150859846 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74dc004eb0 con 0x7f74f80a98f0 2024-03-25T18:05:10.989 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.988+0000 7f74ee7fc640 1 -- 172.21.15.66:0/150859846 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f74f4001970 con 0x7f74f80a98f0 2024-03-25T18:05:10.989 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.988+0000 7f74ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:10.989 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.988+0000 7f74ee7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:10.989 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.988+0000 7f74ee7fc640 10 monclient(hunting): dump: 2024-03-25T18:05:10.989 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:10.989 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:10.989 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:10.990 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:10.990 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:10.990 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:10.990 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:10.990 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:10.990 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:10.990 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:10.991 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.990+0000 7f74ee7fc640 1 -- 172.21.15.66:0/150859846 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 769755571 0 0) 0x7f74f81dc070 con 0x7f74f813f710 2024-03-25T18:05:10.991 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.990+0000 7f74ee7fc640 1 -- 172.21.15.66:0/150859846 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74f8137750 con 0x7f74f813f710 2024-03-25T18:05:10.991 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.990+0000 7f74ee7fc640 1 -- 172.21.15.66:0/150859846 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f74e8001b00 con 0x7f74f813f710 2024-03-25T18:05:10.992 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.990+0000 7f74ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:10.992 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.990+0000 7f74ee7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:05:10.992 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.990+0000 7f74ee7fc640 10 monclient(hunting): dump: 2024-03-25T18:05:10.992 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:10.992 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:10.992 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:10.992 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:10.992 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:10.992 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:10.992 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:10.992 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:10.992 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:10.992 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:10.998 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.996+0000 7f74ee7fc640 1 -- 172.21.15.66:0/150859846 <== mon.2 v1:172.21.15.66:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 971270902 0 0) 0x7f74f8137750 con 0x7f74f813f710 2024-03-25T18:05:10.998 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.996+0000 7f74ee7fc640 1 -- 172.21.15.66:0/150859846 >> v1:172.21.15.193:6789/0 conn(0x7f74f81430c0 legacy=0x7f74f81db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:10.998 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.996+0000 7f74ee7fc640 1 -- 172.21.15.66:0/150859846 >> v1:172.21.15.66:6789/0 conn(0x7f74f80a98f0 legacy=0x7f74f8137c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:10.998 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.996+0000 7f74ee7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:05:10.998 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.996+0000 7f74ee7fc640 1 monclient: found mon.c 2024-03-25T18:05:10.998 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.996+0000 7f74ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:10.998 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.996+0000 7f74ee7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:05:10.998 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.996+0000 7f74ee7fc640 1 -- 172.21.15.66:0/150859846 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f8138c70 con 0x7f74f813f710 2024-03-25T18:05:10.998 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.996+0000 7f74ee7fc640 10 monclient: _finish_auth 0 2024-03-25T18:05:10.999 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.996+0000 7f74ee7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:05:10.999 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.996+0000 7f74ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:10.999 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.996+0000 7f7500250640 5 monclient: authenticate success, global_id 4283 2024-03-25T18:05:10.999 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.996+0000 7f7500250640 10 monclient: _renew_subs 2024-03-25T18:05:10.999 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.996+0000 7f7500250640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:05:10.999 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.996+0000 7f7500250640 1 -- 172.21.15.66:0/150859846 --> v1:172.21.15.66:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74f8138f50 con 0x7f74f813f710 2024-03-25T18:05:10.999 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.997+0000 7f7500250640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:10.999 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.997+0000 7f7500250640 10 monclient: _renew_subs 2024-03-25T18:05:10.999 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.997+0000 7f7500250640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:05:10.999 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.997+0000 7f7500250640 1 -- 172.21.15.66:0/150859846 --> v1:172.21.15.66:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f8139510 con 0x7f74f813f710 2024-03-25T18:05:10.999 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.997+0000 7f7500250640 1 librados: init done 2024-03-25T18:05:11.000 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.998+0000 7f7500250640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:11.000 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.998+0000 7f7500250640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:11.000 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.998+0000 7f7500250640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:05:11.000 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:10.998+0000 7f7500250640 1 -- 172.21.15.66:0/150859846 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74f81378e0 con 0x7f74f813f710 2024-03-25T18:05:11.309 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.308+0000 7f74ee7fc640 1 -- 172.21.15.66:0/150859846 <== mon.2 v1:172.21.15.66:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74e8001210 con 0x7f74f813f710 2024-03-25T18:05:11.310 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.308+0000 7f74ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:11.310 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.308+0000 7f74ee7fc640 1 -- 172.21.15.66:0/150859846 <== mon.2 v1:172.21.15.66:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f74e8004a90 con 0x7f74f813f710 2024-03-25T18:05:11.310 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.308+0000 7f74ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:11.310 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.308+0000 7f74ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:05:11.310 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.308+0000 7f74ee7fc640 10 monclient: dump: 2024-03-25T18:05:11.310 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:11.310 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:11.310 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:11.310 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:11.310 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:11.310 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:11.310 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:11.311 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:11.311 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:11.311 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:11.311 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.308+0000 7f74effff640 4 set_mon_vals no callback set 2024-03-25T18:05:11.311 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.309+0000 7f74ee7fc640 1 -- 172.21.15.66:0/150859846 <== mon.2 v1:172.21.15.66:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f74e801aa60 con 0x7f74f813f710 2024-03-25T18:05:11.311 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.310+0000 7f74fdfec640 10 monclient: get_auth_request con 0x7f74dc06e110 auth_method 0 2024-03-25T18:05:11.312 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.311+0000 7f74ee7fc640 10 client.4283.objecter ms_handle_connect 0x7f74dc06e110 2024-03-25T18:05:11.317 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.315+0000 7f74ee7fc640 1 -- 172.21.15.66:0/150859846 <== mon.2 v1:172.21.15.66:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f74e8054680 con 0x7f74f813f710 2024-03-25T18:05:11.317 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.315+0000 7f74ee7fc640 10 client.4283.objecter ms_dispatch 0x7f74f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:05:11.317 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.315+0000 7f74ee7fc640 3 client.4283.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:05:11.317 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.315+0000 7f74ee7fc640 3 client.4283.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:05:11.317 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.315+0000 7f74ee7fc640 20 client.4283.objecter dump_active .. 0 homeless 2024-03-25T18:05:11.317 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.315+0000 7f74ee7fc640 1 -- 172.21.15.66:0/150859846 <== mon.2 v1:172.21.15.66:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f74e804ffc0 con 0x7f74f813f710 2024-03-25T18:05:11.317 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.315+0000 7f74ee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:11.317 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.315+0000 7f74ee7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:05:11.575 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.574+0000 7f7500250640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:11.575 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.574+0000 7f7500250640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:11.575 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.574+0000 7f7500250640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:05:11.575 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.574+0000 7f7500250640 1 -- 172.21.15.66:0/150859846 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74f80627b0 con 0x7f74f813f710 2024-03-25T18:05:11.601 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.600+0000 7f74ee7fc640 1 -- 172.21.15.66:0/150859846 <== mon.2 v1:172.21.15.66:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f74e80520c0 con 0x7f74f813f710 2024-03-25T18:05:11.601 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.600+0000 7f74ee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:11.601 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.600+0000 7f74ee7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:05:11.602 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:05:11.602 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:05:11.605 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.603+0000 7f7500250640 10 librados: watch_flush enter 2024-03-25T18:05:11.605 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.603+0000 7f7500250640 10 librados: watch_flush exit 2024-03-25T18:05:11.605 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.603+0000 7f7500250640 20 client.4283.objecter shutdown clearing up homeless session... 2024-03-25T18:05:11.605 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.603+0000 7f7500250640 10 client.4283.objecter successfully canceled tick 2024-03-25T18:05:11.605 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.603+0000 7f7500250640 1 -- 172.21.15.66:0/150859846 >> v1:172.21.15.193:6800/2275676803 conn(0x7f74dc06e110 legacy=0x7f74dc0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:11.605 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.603+0000 7f7500250640 10 monclient: shutdown 2024-03-25T18:05:11.605 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.603+0000 7f7500250640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:11.605 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.603+0000 7f7500250640 1 -- 172.21.15.66:0/150859846 >> v1:172.21.15.66:6790/0 conn(0x7f74f813f710 legacy=0x7f74f81db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:11.605 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.603+0000 7f7500250640 1 -- 172.21.15.66:0/150859846 shutdown_connections 2024-03-25T18:05:11.605 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.603+0000 7f7500250640 1 -- 172.21.15.66:0/150859846 >> 172.21.15.66:0/150859846 conn(0x7f74f80b7ea0 msgr2=0x7f74f8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:11.606 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.603+0000 7f7500250640 1 -- 172.21.15.66:0/150859846 shutdown_connections 2024-03-25T18:05:11.606 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.603+0000 7f7500250640 1 -- 172.21.15.66:0/150859846 wait complete. 2024-03-25T18:05:11.606 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:11.603+0000 7f7500250640 1 librados: shutdown 2024-03-25T18:05:11.616 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:05:18.422 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:05:18.539 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.538+0000 7f2d9f49a640 10 monclient: get_monmap_and_config 2024-03-25T18:05:18.539 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.538+0000 7f2d9f49a640 10 monclient: build_initial_monmap 2024-03-25T18:05:18.539 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.538+0000 7f2d9f49a640 10 monclient: monmap: 2024-03-25T18:05:18.539 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:05:18.540 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:18.540 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:05:18.539908+0000 2024-03-25T18:05:18.540 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:05:18.539908+0000 2024-03-25T18:05:18.540 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:18.540 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:18.540 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:05:18.540 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:05:18.540 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:05:18.540 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:18.541 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.539+0000 7f2d9f49a640 1 Processor -- start 2024-03-25T18:05:18.541 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.539+0000 7f2d9f49a640 1 -- start start 2024-03-25T18:05:18.541 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.539+0000 7f2d9f49a640 10 monclient: init 2024-03-25T18:05:18.542 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.540+0000 7f2d9f49a640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:18.542 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.540+0000 7f2d9f49a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:05:18.542 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.540+0000 7f2d9f49a640 10 monclient(hunting): picked mon.noname-c con 0x7f2d980b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T18:05:18.542 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.540+0000 7f2d9f49a640 10 monclient(hunting): picked mon.noname-a con 0x7f2d980b1730 addr v1:172.21.15.66:6789/0 2024-03-25T18:05:18.542 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.540+0000 7f2d9f49a640 10 monclient(hunting): picked mon.noname-b con 0x7f2d9814be90 addr v1:172.21.15.66:6790/0 2024-03-25T18:05:18.542 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.540+0000 7f2d9f49a640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d980af940 con 0x7f2d980b1730 2024-03-25T18:05:18.542 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.540+0000 7f2d9f49a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d980b0a70 con 0x7f2d980b4d60 2024-03-25T18:05:18.542 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.540+0000 7f2d9f49a640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d980b0d50 con 0x7f2d9814be90 2024-03-25T18:05:18.542 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.540+0000 7f2d9f49a640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:18.542 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.540+0000 7f2d9f49a640 10 monclient(hunting): authenticate will time out at 7067.600309s 2024-03-25T18:05:18.542 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.540+0000 7f2d9d236640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f2d980b4d60 0x7f2d9813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:40660/0 (socket says 172.21.15.66:40660) 2024-03-25T18:05:18.542 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.540+0000 7f2d9d236640 1 -- 172.21.15.66:0/2094161894 learned_addr learned my addr 172.21.15.66:0/2094161894 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:05:18.631 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.629+0000 7f2d8ffff640 1 -- 172.21.15.66:0/2094161894 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478236464 0 0) 0x7f2d980af940 con 0x7f2d980b1730 2024-03-25T18:05:18.631 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.629+0000 7f2d8ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:05:18.631 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.629+0000 7f2d8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:18.631 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.629+0000 7f2d8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:18.631 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.629+0000 7f2d8ffff640 10 monclient(hunting): my global_id is 4287 2024-03-25T18:05:18.631 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.629+0000 7f2d8ffff640 1 -- 172.21.15.66:0/2094161894 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d84003520 con 0x7f2d980b1730 2024-03-25T18:05:18.650 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.648+0000 7f2d8ffff640 1 -- 172.21.15.66:0/2094161894 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1564722823 0 0) 0x7f2d84003520 con 0x7f2d980b1730 2024-03-25T18:05:18.650 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.648+0000 7f2d8ffff640 1 -- 172.21.15.66:0/2094161894 >> v1:172.21.15.66:6790/0 conn(0x7f2d9814be90 legacy=0x7f2d9814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:18.650 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.648+0000 7f2d8ffff640 1 -- 172.21.15.66:0/2094161894 >> v1:172.21.15.193:6789/0 conn(0x7f2d980b4d60 legacy=0x7f2d9813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:18.650 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.649+0000 7f2d8ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:05:18.650 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.649+0000 7f2d8ffff640 1 monclient: found mon.noname-a 2024-03-25T18:05:18.650 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.649+0000 7f2d8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:18.650 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.649+0000 7f2d8ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.66:6789/0 2024-03-25T18:05:18.650 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.649+0000 7f2d8ffff640 1 -- 172.21.15.66:0/2094161894 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d980b1030 con 0x7f2d980b1730 2024-03-25T18:05:18.650 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.649+0000 7f2d8ffff640 10 monclient: _finish_auth 0 2024-03-25T18:05:18.651 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.649+0000 7f2d8ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:05:18.651 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.649+0000 7f2d8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:18.651 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.649+0000 7f2d8ffff640 1 -- 172.21.15.66:0/2094161894 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f2d88001690 con 0x7f2d980b1730 2024-03-25T18:05:18.651 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.649+0000 7f2d9f49a640 5 monclient: authenticate success, global_id 4287 2024-03-25T18:05:18.651 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.649+0000 7f2d9f49a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:18.651 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.649+0000 7f2d8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:18.651 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.649+0000 7f2d8ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:05:18.651 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.649+0000 7f2d8ffff640 10 monclient: dump: 2024-03-25T18:05:18.651 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:18.651 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:18.651 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:18.651 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:18.651 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:18.651 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:18.651 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:18.651 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:18.651 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:18.652 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:18.652 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.649+0000 7f2d9f49a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:18.668 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.666+0000 7f2d8ffff640 1 -- 172.21.15.66:0/2094161894 <== mon.0 v1:172.21.15.66:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d880017f0 con 0x7f2d980b1730 2024-03-25T18:05:18.668 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.666+0000 7f2d8ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:18.668 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.666+0000 7f2d8ffff640 1 -- 172.21.15.66:0/2094161894 <== mon.0 v1:172.21.15.66:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f2d88001c00 con 0x7f2d980b1730 2024-03-25T18:05:18.668 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.666+0000 7f2d9f49a640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:18.668 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.666+0000 7f2d9f49a640 4 set_mon_vals no callback set 2024-03-25T18:05:18.668 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.666+0000 7f2d8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:18.668 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.666+0000 7f2d8ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:18.668 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.666+0000 7f2d8ffff640 10 monclient: dump: 2024-03-25T18:05:18.669 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:18.669 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:18.669 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:18.669 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:18.669 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:18.669 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:18.669 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:18.669 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:18.669 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:18.669 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:18.671 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.667+0000 7f2d9f49a640 10 monclient: shutdown 2024-03-25T18:05:18.671 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.667+0000 7f2d9f49a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:18.671 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.667+0000 7f2d9f49a640 1 -- 172.21.15.66:0/2094161894 >> v1:172.21.15.66:6789/0 conn(0x7f2d980b1730 legacy=0x7f2d980b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:18.671 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.667+0000 7f2d9f49a640 1 -- 172.21.15.66:0/2094161894 shutdown_connections 2024-03-25T18:05:18.671 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.667+0000 7f2d9f49a640 1 -- 172.21.15.66:0/2094161894 >> 172.21.15.66:0/2094161894 conn(0x7f2d980617a0 msgr2=0x7f2d98061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:18.671 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.667+0000 7f2d9f49a640 1 -- 172.21.15.66:0/2094161894 shutdown_connections 2024-03-25T18:05:18.672 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.668+0000 7f2d9f49a640 1 -- 172.21.15.66:0/2094161894 wait complete. 2024-03-25T18:05:18.672 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.668+0000 7f2d9f49a640 10 monclient: build_initial_monmap 2024-03-25T18:05:18.672 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.668+0000 7f2d9f49a640 10 monclient: monmap: 2024-03-25T18:05:18.672 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:05:18.672 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:18.672 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:05:18.672 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:05:18.672 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:18.672 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:05:18.672 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:05:18.672 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:05:18.672 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:05:18.672 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:18.672 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.668+0000 7f2d9f49a640 1 librados: starting msgr at 2024-03-25T18:05:18.672 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.668+0000 7f2d9f49a640 1 librados: starting objecter 2024-03-25T18:05:18.672 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.669+0000 7f2d9f49a640 1 Processor -- start 2024-03-25T18:05:18.672 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.669+0000 7f2d9f49a640 1 -- start start 2024-03-25T18:05:18.673 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.669+0000 7f2d9f49a640 1 librados: setting wanted keys 2024-03-25T18:05:18.673 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.669+0000 7f2d9f49a640 1 librados: calling monclient init 2024-03-25T18:05:18.673 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.669+0000 7f2d9f49a640 10 monclient: init 2024-03-25T18:05:18.673 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.669+0000 7f2d9f49a640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:18.673 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.669+0000 7f2d9f49a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:05:18.673 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.669+0000 7f2d9f49a640 10 monclient(hunting): picked mon.noname-a con 0x7f2d980b1730 addr v1:172.21.15.66:6789/0 2024-03-25T18:05:18.673 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.670+0000 7f2d9f49a640 10 monclient(hunting): picked mon.noname-c con 0x7f2d980b4d60 addr v1:172.21.15.66:6790/0 2024-03-25T18:05:18.673 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.670+0000 7f2d9f49a640 10 monclient(hunting): picked mon.noname-b con 0x7f2d9814be90 addr v1:172.21.15.193:6789/0 2024-03-25T18:05:18.673 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.670+0000 7f2d9f49a640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d98136e40 con 0x7f2d980b1730 2024-03-25T18:05:18.673 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.670+0000 7f2d9f49a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d981307c0 con 0x7f2d9814be90 2024-03-25T18:05:18.673 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.670+0000 7f2d9f49a640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d98138610 con 0x7f2d980b4d60 2024-03-25T18:05:18.673 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.670+0000 7f2d9f49a640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:18.673 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.670+0000 7f2d9f49a640 10 monclient(hunting): authenticate will time out at 7067.730155s 2024-03-25T18:05:18.686 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.685+0000 7f2d9ca35640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f2d980b4d60 0x7f2d9812f9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:51344/0 (socket says 172.21.15.66:51344) 2024-03-25T18:05:18.686 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.685+0000 7f2d9ca35640 1 -- 172.21.15.66:0/2080283299 learned_addr learned my addr 172.21.15.66:0/2080283299 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:05:18.686 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.685+0000 7f2d8dffb640 10 client.?.objecter ms_handle_connect 0x7f2d9814be90 2024-03-25T18:05:18.686 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.685+0000 7f2d8dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:18.687 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.685+0000 7f2d8dffb640 10 client.?.objecter ms_handle_connect 0x7f2d980b4d60 2024-03-25T18:05:18.687 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.686+0000 7f2d8dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:18.717 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.715+0000 7f2d8dffb640 10 client.?.objecter ms_handle_connect 0x7f2d980b1730 2024-03-25T18:05:18.717 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.715+0000 7f2d8dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:18.780 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.779+0000 7f2d8dffb640 1 -- 172.21.15.66:0/2080283299 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076645328 0 0) 0x7f2d98136e40 con 0x7f2d980b1730 2024-03-25T18:05:18.780 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.779+0000 7f2d8dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:05:18.780 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.779+0000 7f2d8dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:18.780 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.779+0000 7f2d8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:18.780 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.779+0000 7f2d8dffb640 10 monclient(hunting): my global_id is 4290 2024-03-25T18:05:18.780 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.779+0000 7f2d8dffb640 1 -- 172.21.15.66:0/2080283299 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d7c004eb0 con 0x7f2d980b1730 2024-03-25T18:05:18.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.790+0000 7f2d8dffb640 1 -- 172.21.15.66:0/2080283299 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1980557045 0 0) 0x7f2d7c004eb0 con 0x7f2d980b1730 2024-03-25T18:05:18.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.790+0000 7f2d8dffb640 1 -- 172.21.15.66:0/2080283299 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d98136e40 con 0x7f2d980b1730 2024-03-25T18:05:18.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.790+0000 7f2d8dffb640 1 -- 172.21.15.66:0/2080283299 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f2d940018a0 con 0x7f2d980b1730 2024-03-25T18:05:18.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.790+0000 7f2d8dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:18.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.790+0000 7f2d8dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:05:18.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.790+0000 7f2d8dffb640 10 monclient(hunting): dump: 2024-03-25T18:05:18.792 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:18.792 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:18.792 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:18.792 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:18.792 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:18.792 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:18.792 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:18.792 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:18.792 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:18.792 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:18.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.790+0000 7f2d8dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:18.793 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.791+0000 7f2d8dffb640 1 -- 172.21.15.66:0/2080283299 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2449595636 0 0) 0x7f2d981307c0 con 0x7f2d9814be90 2024-03-25T18:05:18.793 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.791+0000 7f2d8dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:05:18.793 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.791+0000 7f2d8dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:18.793 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.791+0000 7f2d8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:18.793 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.791+0000 7f2d8dffb640 10 monclient(hunting): my global_id is 4288 2024-03-25T18:05:18.793 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.791+0000 7f2d8dffb640 1 -- 172.21.15.66:0/2080283299 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d7c004eb0 con 0x7f2d9814be90 2024-03-25T18:05:18.793 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.791+0000 7f2d8dffb640 1 -- 172.21.15.66:0/2080283299 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 981451405 0 0) 0x7f2d98138610 con 0x7f2d980b4d60 2024-03-25T18:05:18.793 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.791+0000 7f2d8dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:05:18.793 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.791+0000 7f2d8dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:18.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.791+0000 7f2d8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:18.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.791+0000 7f2d8dffb640 10 monclient(hunting): my global_id is 4286 2024-03-25T18:05:18.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.791+0000 7f2d8dffb640 1 -- 172.21.15.66:0/2080283299 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d981307c0 con 0x7f2d980b4d60 2024-03-25T18:05:18.806 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.805+0000 7f2d8dffb640 1 -- 172.21.15.66:0/2080283299 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2929558337 0 0) 0x7f2d7c004eb0 con 0x7f2d9814be90 2024-03-25T18:05:18.806 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.805+0000 7f2d8dffb640 1 -- 172.21.15.66:0/2080283299 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d98138610 con 0x7f2d9814be90 2024-03-25T18:05:18.806 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.805+0000 7f2d8dffb640 1 -- 172.21.15.66:0/2080283299 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f2d90001a60 con 0x7f2d9814be90 2024-03-25T18:05:18.807 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.805+0000 7f2d8dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:18.807 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.805+0000 7f2d8dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:18.807 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.805+0000 7f2d8dffb640 10 monclient(hunting): dump: 2024-03-25T18:05:18.807 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:18.807 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:18.807 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:18.807 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:18.807 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:18.807 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:18.807 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:18.807 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:18.807 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:18.807 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:18.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.813+0000 7f2d8dffb640 1 -- 172.21.15.66:0/2080283299 <== mon.0 v1:172.21.15.66:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2965302416 0 0) 0x7f2d98136e40 con 0x7f2d980b1730 2024-03-25T18:05:18.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.813+0000 7f2d8dffb640 1 -- 172.21.15.66:0/2080283299 >> v1:172.21.15.66:6790/0 conn(0x7f2d980b4d60 legacy=0x7f2d9812f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:18.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.813+0000 7f2d8dffb640 1 -- 172.21.15.66:0/2080283299 >> v1:172.21.15.193:6789/0 conn(0x7f2d9814be90 legacy=0x7f2d981300b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:18.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.813+0000 7f2d8dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:05:18.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.813+0000 7f2d8dffb640 1 monclient: found mon.a 2024-03-25T18:05:18.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.813+0000 7f2d8dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:18.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.813+0000 7f2d8dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:05:18.816 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.813+0000 7f2d8dffb640 1 -- 172.21.15.66:0/2080283299 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d981388f0 con 0x7f2d980b1730 2024-03-25T18:05:18.816 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.813+0000 7f2d8dffb640 10 monclient: _finish_auth 0 2024-03-25T18:05:18.816 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.813+0000 7f2d8dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:05:18.816 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.813+0000 7f2d8dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:18.816 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.813+0000 7f2d9f49a640 5 monclient: authenticate success, global_id 4290 2024-03-25T18:05:18.816 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.813+0000 7f2d9f49a640 10 monclient: _renew_subs 2024-03-25T18:05:18.816 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.814+0000 7f2d9f49a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:05:18.816 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.814+0000 7f2d9f49a640 1 -- 172.21.15.66:0/2080283299 --> v1:172.21.15.66:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d981410e0 con 0x7f2d980b1730 2024-03-25T18:05:18.816 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.814+0000 7f2d9f49a640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:18.816 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.814+0000 7f2d9f49a640 10 monclient: _renew_subs 2024-03-25T18:05:18.816 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.814+0000 7f2d9f49a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:05:18.816 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.814+0000 7f2d9f49a640 1 -- 172.21.15.66:0/2080283299 --> v1:172.21.15.66:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d981381a0 con 0x7f2d980b1730 2024-03-25T18:05:18.816 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.814+0000 7f2d9f49a640 1 librados: init done 2024-03-25T18:05:18.817 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.815+0000 7f2d9f49a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:18.817 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.815+0000 7f2d9f49a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:18.817 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.815+0000 7f2d9f49a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:05:18.817 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:18.815+0000 7f2d9f49a640 1 -- 172.21.15.66:0/2080283299 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d981401f0 con 0x7f2d980b1730 2024-03-25T18:05:19.142 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.140+0000 7f2d8dffb640 1 -- 172.21.15.66:0/2080283299 <== mon.0 v1:172.21.15.66:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d94001d50 con 0x7f2d980b1730 2024-03-25T18:05:19.142 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.140+0000 7f2d8dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:19.142 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.140+0000 7f2d8dffb640 1 -- 172.21.15.66:0/2080283299 <== mon.0 v1:172.21.15.66:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f2d94004a40 con 0x7f2d980b1730 2024-03-25T18:05:19.142 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.140+0000 7f2d8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:19.142 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.140+0000 7f2d8dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:19.142 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.140+0000 7f2d8dffb640 10 monclient: dump: 2024-03-25T18:05:19.142 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:19.142 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:19.142 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:19.142 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:19.142 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:19.143 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:19.143 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:19.143 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:19.143 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:19.143 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:19.143 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.140+0000 7f2d8f7fe640 4 set_mon_vals no callback set 2024-03-25T18:05:19.143 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.141+0000 7f2d8dffb640 1 -- 172.21.15.66:0/2080283299 <== mon.0 v1:172.21.15.66:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f2d94004c60 con 0x7f2d980b1730 2024-03-25T18:05:19.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.142+0000 7f2d9ca35640 10 monclient: get_auth_request con 0x7f2d7c06e020 auth_method 0 2024-03-25T18:05:19.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.142+0000 7f2d8dffb640 10 client.4290.objecter ms_handle_connect 0x7f2d7c06e020 2024-03-25T18:05:19.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.148+0000 7f2d8dffb640 1 -- 172.21.15.66:0/2080283299 <== mon.0 v1:172.21.15.66:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f2d940547e0 con 0x7f2d980b1730 2024-03-25T18:05:19.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.148+0000 7f2d8dffb640 10 client.4290.objecter ms_dispatch 0x7f2d98000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:05:19.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.148+0000 7f2d8dffb640 3 client.4290.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:05:19.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.148+0000 7f2d8dffb640 3 client.4290.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:05:19.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.148+0000 7f2d8dffb640 20 client.4290.objecter dump_active .. 0 homeless 2024-03-25T18:05:19.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.148+0000 7f2d8dffb640 1 -- 172.21.15.66:0/2080283299 <== mon.0 v1:172.21.15.66:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2d94050120 con 0x7f2d980b1730 2024-03-25T18:05:19.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.148+0000 7f2d8dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:19.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.148+0000 7f2d8dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:05:19.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.410+0000 7f2d9f49a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:19.412 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.410+0000 7f2d9f49a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:19.412 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.410+0000 7f2d9f49a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:05:19.412 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.410+0000 7f2d9f49a640 1 -- 172.21.15.66:0/2080283299 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d980627b0 con 0x7f2d980b1730 2024-03-25T18:05:19.449 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.448+0000 7f2d8dffb640 1 -- 172.21.15.66:0/2080283299 <== mon.0 v1:172.21.15.66:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f2d94052220 con 0x7f2d980b1730 2024-03-25T18:05:19.449 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.448+0000 7f2d8dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:19.449 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.448+0000 7f2d8dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:05:19.450 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:05:19.450 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:05:19.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.450+0000 7f2d9f49a640 10 librados: watch_flush enter 2024-03-25T18:05:19.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.450+0000 7f2d9f49a640 10 librados: watch_flush exit 2024-03-25T18:05:19.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.450+0000 7f2d9f49a640 20 client.4290.objecter shutdown clearing up homeless session... 2024-03-25T18:05:19.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.450+0000 7f2d9f49a640 10 client.4290.objecter successfully canceled tick 2024-03-25T18:05:19.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.450+0000 7f2d9f49a640 1 -- 172.21.15.66:0/2080283299 >> v1:172.21.15.193:6800/2275676803 conn(0x7f2d7c06e020 legacy=0x7f2d7c0704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:19.453 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.450+0000 7f2d9f49a640 10 monclient: shutdown 2024-03-25T18:05:19.453 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.450+0000 7f2d9f49a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:19.453 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.450+0000 7f2d9f49a640 1 -- 172.21.15.66:0/2080283299 >> v1:172.21.15.66:6789/0 conn(0x7f2d980b1730 legacy=0x7f2d981409d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:19.453 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.450+0000 7f2d9f49a640 1 -- 172.21.15.66:0/2080283299 shutdown_connections 2024-03-25T18:05:19.453 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.450+0000 7f2d9f49a640 1 -- 172.21.15.66:0/2080283299 >> 172.21.15.66:0/2080283299 conn(0x7f2d980617a0 msgr2=0x7f2d980b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:19.453 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.451+0000 7f2d9f49a640 1 -- 172.21.15.66:0/2080283299 shutdown_connections 2024-03-25T18:05:19.453 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.451+0000 7f2d9f49a640 1 -- 172.21.15.66:0/2080283299 wait complete. 2024-03-25T18:05:19.453 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:19.451+0000 7f2d9f49a640 1 librados: shutdown 2024-03-25T18:05:19.464 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:05:23.754 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:01:54.735 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:23.765 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:01:54.745 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:23.810 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:01:54.788 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:23.819 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:01:54.778 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:23.821 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:01:54.802 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:23.824 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:01:54.783 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:23.830 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:01:54.811 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:23.854 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:01:54.835 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:23.857 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:01:54.816 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:26.272 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:05:26.376 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.374+0000 7f047c739640 10 monclient: get_monmap_and_config 2024-03-25T18:05:26.376 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.374+0000 7f047c739640 10 monclient: build_initial_monmap 2024-03-25T18:05:26.376 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.374+0000 7f047c739640 10 monclient: monmap: 2024-03-25T18:05:26.376 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:05:26.376 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:26.376 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:05:26.376272+0000 2024-03-25T18:05:26.376 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:05:26.376272+0000 2024-03-25T18:05:26.376 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:26.376 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:26.376 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:05:26.376 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:05:26.376 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:05:26.377 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:26.378 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.375+0000 7f047c739640 1 Processor -- start 2024-03-25T18:05:26.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.375+0000 7f047c739640 1 -- start start 2024-03-25T18:05:26.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.375+0000 7f047c739640 10 monclient: init 2024-03-25T18:05:26.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.376+0000 7f047c739640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:26.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.376+0000 7f047c739640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:05:26.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.376+0000 7f047c739640 10 monclient(hunting): picked mon.noname-b con 0x7f04740b4d60 addr v1:172.21.15.66:6790/0 2024-03-25T18:05:26.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.376+0000 7f047c739640 10 monclient(hunting): picked mon.noname-a con 0x7f04740b1730 addr v1:172.21.15.66:6789/0 2024-03-25T18:05:26.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.376+0000 7f047c739640 10 monclient(hunting): picked mon.noname-c con 0x7f047414be90 addr v1:172.21.15.193:6789/0 2024-03-25T18:05:26.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.376+0000 7f047c739640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04740af940 con 0x7f04740b1730 2024-03-25T18:05:26.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.376+0000 7f047c739640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04740b0a70 con 0x7f047414be90 2024-03-25T18:05:26.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.376+0000 7f047c739640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04740b0d50 con 0x7f04740b4d60 2024-03-25T18:05:26.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.376+0000 7f047c739640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:26.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.376+0000 7f047c739640 10 monclient(hunting): authenticate will time out at 7075.436403s 2024-03-25T18:05:26.380 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.376+0000 7f047acd6640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f047414be90 0x7f047414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:33558/0 (socket says 172.21.15.66:33558) 2024-03-25T18:05:26.380 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.376+0000 7f047acd6640 1 -- 172.21.15.66:0/926758661 learned_addr learned my addr 172.21.15.66:0/926758661 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:05:26.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.406+0000 7f04794d3640 1 -- 172.21.15.66:0/926758661 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 147850086 0 0) 0x7f04740b0d50 con 0x7f04740b4d60 2024-03-25T18:05:26.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.406+0000 7f04794d3640 10 monclient(hunting): _negotiate 2024-03-25T18:05:26.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.406+0000 7f04794d3640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:26.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.406+0000 7f04794d3640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:26.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.406+0000 7f04794d3640 10 monclient(hunting): my global_id is 4289 2024-03-25T18:05:26.408 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.406+0000 7f04794d3640 1 -- 172.21.15.66:0/926758661 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0460003520 con 0x7f04740b4d60 2024-03-25T18:05:26.423 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.421+0000 7f04794d3640 1 -- 172.21.15.66:0/926758661 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3603921328 0 0) 0x7f0460003520 con 0x7f04740b4d60 2024-03-25T18:05:26.423 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.421+0000 7f04794d3640 1 -- 172.21.15.66:0/926758661 >> v1:172.21.15.193:6789/0 conn(0x7f047414be90 legacy=0x7f047414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:26.423 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.421+0000 7f04794d3640 1 -- 172.21.15.66:0/926758661 >> v1:172.21.15.66:6789/0 conn(0x7f04740b1730 legacy=0x7f04740b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:26.423 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.421+0000 7f04794d3640 10 monclient: _finish_hunting 0 2024-03-25T18:05:26.423 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.421+0000 7f04794d3640 1 monclient: found mon.noname-b 2024-03-25T18:05:26.423 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.421+0000 7f04794d3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:26.423 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.421+0000 7f04794d3640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.66:6790/0 2024-03-25T18:05:26.423 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.421+0000 7f04794d3640 1 -- 172.21.15.66:0/926758661 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04740b1030 con 0x7f04740b4d60 2024-03-25T18:05:26.423 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.421+0000 7f04794d3640 10 monclient: _finish_auth 0 2024-03-25T18:05:26.423 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.421+0000 7f04794d3640 10 monclient: _check_auth_tickets 2024-03-25T18:05:26.424 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.421+0000 7f04794d3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:26.424 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.421+0000 7f04794d3640 1 -- 172.21.15.66:0/926758661 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f0470001690 con 0x7f04740b4d60 2024-03-25T18:05:26.424 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.421+0000 7f04794d3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:26.424 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.421+0000 7f04794d3640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:05:26.424 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.421+0000 7f04794d3640 10 monclient: dump: 2024-03-25T18:05:26.424 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:26.424 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:26.424 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:26.424 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:26.424 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:26.424 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:26.424 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:26.424 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:26.424 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:26.425 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:26.425 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.421+0000 7f047c739640 5 monclient: authenticate success, global_id 4289 2024-03-25T18:05:26.425 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.421+0000 7f047c739640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:26.441 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.440+0000 7f04794d3640 1 -- 172.21.15.66:0/926758661 <== mon.2 v1:172.21.15.66:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04700017f0 con 0x7f04740b4d60 2024-03-25T18:05:26.441 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.440+0000 7f04794d3640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:26.441 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.440+0000 7f04794d3640 1 -- 172.21.15.66:0/926758661 <== mon.2 v1:172.21.15.66:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f0470001c00 con 0x7f04740b4d60 2024-03-25T18:05:26.441 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.440+0000 7f047c739640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:26.441 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.440+0000 7f047c739640 4 set_mon_vals no callback set 2024-03-25T18:05:26.441 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.440+0000 7f04794d3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:26.441 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.440+0000 7f04794d3640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:05:26.441 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.440+0000 7f04794d3640 10 monclient: dump: 2024-03-25T18:05:26.442 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:26.442 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:26.442 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:26.442 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:26.442 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:26.442 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:26.442 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:26.442 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:26.442 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:26.442 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:26.445 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.440+0000 7f047c739640 10 monclient: shutdown 2024-03-25T18:05:26.445 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.440+0000 7f047c739640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:26.445 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.440+0000 7f047c739640 1 -- 172.21.15.66:0/926758661 >> v1:172.21.15.66:6790/0 conn(0x7f04740b4d60 legacy=0x7f047413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:26.445 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.440+0000 7f047c739640 1 -- 172.21.15.66:0/926758661 shutdown_connections 2024-03-25T18:05:26.445 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.440+0000 7f047c739640 1 -- 172.21.15.66:0/926758661 >> 172.21.15.66:0/926758661 conn(0x7f04740617a0 msgr2=0x7f0474061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:26.445 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.440+0000 7f047c739640 1 -- 172.21.15.66:0/926758661 shutdown_connections 2024-03-25T18:05:26.445 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.441+0000 7f047c739640 1 -- 172.21.15.66:0/926758661 wait complete. 2024-03-25T18:05:26.445 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.441+0000 7f047c739640 10 monclient: build_initial_monmap 2024-03-25T18:05:26.445 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.441+0000 7f047c739640 10 monclient: monmap: 2024-03-25T18:05:26.445 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:05:26.445 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:26.445 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:05:26.446 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:05:26.446 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:26.446 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:05:26.446 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:05:26.446 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:05:26.446 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:05:26.446 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:26.446 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.441+0000 7f047c739640 1 librados: starting msgr at 2024-03-25T18:05:26.446 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.441+0000 7f047c739640 1 librados: starting objecter 2024-03-25T18:05:26.446 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.442+0000 7f047c739640 1 Processor -- start 2024-03-25T18:05:26.446 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.442+0000 7f047c739640 1 -- start start 2024-03-25T18:05:26.446 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.442+0000 7f047c739640 1 librados: setting wanted keys 2024-03-25T18:05:26.446 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.442+0000 7f047c739640 1 librados: calling monclient init 2024-03-25T18:05:26.446 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.442+0000 7f047c739640 10 monclient: init 2024-03-25T18:05:26.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.442+0000 7f047c739640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:26.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.442+0000 7f047c739640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:05:26.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.442+0000 7f047c739640 10 monclient(hunting): picked mon.noname-b con 0x7f04740b1730 addr v1:172.21.15.193:6789/0 2024-03-25T18:05:26.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.442+0000 7f047c739640 10 monclient(hunting): picked mon.noname-c con 0x7f04740b4d60 addr v1:172.21.15.66:6790/0 2024-03-25T18:05:26.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.442+0000 7f047c739640 10 monclient(hunting): picked mon.noname-a con 0x7f047414be90 addr v1:172.21.15.66:6789/0 2024-03-25T18:05:26.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.442+0000 7f047c739640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04741dbff0 con 0x7f047414be90 2024-03-25T18:05:26.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.442+0000 7f047c739640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04741327f0 con 0x7f04740b1730 2024-03-25T18:05:26.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.442+0000 7f047c739640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0474132f60 con 0x7f04740b4d60 2024-03-25T18:05:26.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.442+0000 7f047c739640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:26.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.442+0000 7f047c739640 10 monclient(hunting): authenticate will time out at 7075.502951s 2024-03-25T18:05:26.453 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.452+0000 7f0479cd4640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f04740b4d60 0x7f0474137130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:45422/0 (socket says 172.21.15.66:45422) 2024-03-25T18:05:26.453 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.452+0000 7f0479cd4640 1 -- 172.21.15.66:0/3387145409 learned_addr learned my addr 172.21.15.66:0/3387145409 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:05:26.456 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.455+0000 7f046affd640 10 client.?.objecter ms_handle_connect 0x7f04740b4d60 2024-03-25T18:05:26.456 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.455+0000 7f046affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:26.463 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.462+0000 7f046affd640 10 client.?.objecter ms_handle_connect 0x7f047414be90 2024-03-25T18:05:26.463 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.462+0000 7f046affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:26.468 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.466+0000 7f046affd640 1 -- 172.21.15.66:0/3387145409 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181701449 0 0) 0x7f0474132f60 con 0x7f04740b4d60 2024-03-25T18:05:26.468 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.466+0000 7f046affd640 10 monclient(hunting): _negotiate 2024-03-25T18:05:26.468 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.466+0000 7f046affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:26.468 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.466+0000 7f046affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:26.468 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.466+0000 7f046affd640 10 monclient(hunting): my global_id is 4292 2024-03-25T18:05:26.468 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.466+0000 7f046affd640 1 -- 172.21.15.66:0/3387145409 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0458004eb0 con 0x7f04740b4d60 2024-03-25T18:05:26.474 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.473+0000 7f046affd640 10 client.?.objecter ms_handle_connect 0x7f04740b1730 2024-03-25T18:05:26.474 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.473+0000 7f046affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:26.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.485+0000 7f046affd640 1 -- 172.21.15.66:0/3387145409 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2736573007 0 0) 0x7f04741327f0 con 0x7f04740b1730 2024-03-25T18:05:26.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.485+0000 7f046affd640 10 monclient(hunting): _negotiate 2024-03-25T18:05:26.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.485+0000 7f046affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:26.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.485+0000 7f046affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:26.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.485+0000 7f046affd640 10 monclient(hunting): my global_id is 4291 2024-03-25T18:05:26.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.485+0000 7f046affd640 1 -- 172.21.15.66:0/3387145409 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0474132f60 con 0x7f04740b1730 2024-03-25T18:05:26.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.490+0000 7f046affd640 1 -- 172.21.15.66:0/3387145409 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 72043402 0 0) 0x7f0458004eb0 con 0x7f04740b4d60 2024-03-25T18:05:26.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.490+0000 7f046affd640 1 -- 172.21.15.66:0/3387145409 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04741327f0 con 0x7f04740b4d60 2024-03-25T18:05:26.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.490+0000 7f046affd640 1 -- 172.21.15.66:0/3387145409 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f0464001a30 con 0x7f04740b4d60 2024-03-25T18:05:26.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.490+0000 7f046affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:26.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.490+0000 7f046affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:05:26.492 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.490+0000 7f046affd640 10 monclient(hunting): dump: 2024-03-25T18:05:26.492 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:26.492 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:26.492 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:26.492 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:26.492 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:26.492 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:26.492 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:26.492 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:26.492 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:26.492 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:26.492 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.490+0000 7f046affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:26.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.495+0000 7f046affd640 1 -- 172.21.15.66:0/3387145409 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 855469741 0 0) 0x7f04741dbff0 con 0x7f047414be90 2024-03-25T18:05:26.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.495+0000 7f046affd640 10 monclient(hunting): _negotiate 2024-03-25T18:05:26.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.495+0000 7f046affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:26.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.495+0000 7f046affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:26.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.495+0000 7f046affd640 10 monclient(hunting): my global_id is 4296 2024-03-25T18:05:26.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.495+0000 7f046affd640 1 -- 172.21.15.66:0/3387145409 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0458004eb0 con 0x7f047414be90 2024-03-25T18:05:26.506 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.505+0000 7f046affd640 1 -- 172.21.15.66:0/3387145409 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1229326291 0 0) 0x7f0474132f60 con 0x7f04740b1730 2024-03-25T18:05:26.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.505+0000 7f046affd640 1 -- 172.21.15.66:0/3387145409 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04741dbff0 con 0x7f04740b1730 2024-03-25T18:05:26.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.505+0000 7f046affd640 1 -- 172.21.15.66:0/3387145409 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f0470001b40 con 0x7f04740b1730 2024-03-25T18:05:26.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.505+0000 7f046affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:26.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.505+0000 7f046affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:26.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.505+0000 7f046affd640 10 monclient(hunting): dump: 2024-03-25T18:05:26.507 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:26.507 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:26.507 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:26.507 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:26.507 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:26.507 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:26.507 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:26.507 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:26.508 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:26.508 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:26.513 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.512+0000 7f046affd640 1 -- 172.21.15.66:0/3387145409 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4220579855 0 0) 0x7f0458004eb0 con 0x7f047414be90 2024-03-25T18:05:26.513 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.512+0000 7f046affd640 1 -- 172.21.15.66:0/3387145409 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0474132f60 con 0x7f047414be90 2024-03-25T18:05:26.513 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.512+0000 7f046affd640 1 -- 172.21.15.66:0/3387145409 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f046c001a60 con 0x7f047414be90 2024-03-25T18:05:26.514 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.512+0000 7f046affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:26.514 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.512+0000 7f046affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:26.514 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.512+0000 7f046affd640 10 monclient(hunting): dump: 2024-03-25T18:05:26.514 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:26.514 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:26.514 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:26.514 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:26.514 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:26.514 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:26.514 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:26.514 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:26.514 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:26.514 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:26.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.513+0000 7f046affd640 1 -- 172.21.15.66:0/3387145409 <== mon.2 v1:172.21.15.66:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 780589792 0 0) 0x7f04741327f0 con 0x7f04740b4d60 2024-03-25T18:05:26.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.513+0000 7f046affd640 1 -- 172.21.15.66:0/3387145409 >> v1:172.21.15.193:6789/0 conn(0x7f04740b1730 legacy=0x7f047412da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:26.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.513+0000 7f046affd640 1 -- 172.21.15.66:0/3387145409 >> v1:172.21.15.66:6789/0 conn(0x7f047414be90 legacy=0x7f04741e33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:26.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.513+0000 7f046affd640 10 monclient: _finish_hunting 0 2024-03-25T18:05:26.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.513+0000 7f046affd640 1 monclient: found mon.c 2024-03-25T18:05:26.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.513+0000 7f046affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:26.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.513+0000 7f046affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:05:26.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.513+0000 7f046affd640 1 -- 172.21.15.66:0/3387145409 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f047412f5b0 con 0x7f04740b4d60 2024-03-25T18:05:26.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.513+0000 7f046affd640 10 monclient: _finish_auth 0 2024-03-25T18:05:26.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.513+0000 7f046affd640 10 monclient: _check_auth_tickets 2024-03-25T18:05:26.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.513+0000 7f046affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:26.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.513+0000 7f047c739640 5 monclient: authenticate success, global_id 4292 2024-03-25T18:05:26.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.513+0000 7f047c739640 10 monclient: _renew_subs 2024-03-25T18:05:26.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.513+0000 7f047c739640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:05:26.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.513+0000 7f047c739640 1 -- 172.21.15.66:0/3387145409 --> v1:172.21.15.66:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f047412f900 con 0x7f04740b4d60 2024-03-25T18:05:26.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.513+0000 7f047c739640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:26.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.513+0000 7f047c739640 10 monclient: _renew_subs 2024-03-25T18:05:26.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.513+0000 7f047c739640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:05:26.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.513+0000 7f047c739640 1 -- 172.21.15.66:0/3387145409 --> v1:172.21.15.66:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04741db090 con 0x7f04740b4d60 2024-03-25T18:05:26.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.513+0000 7f047c739640 1 librados: init done 2024-03-25T18:05:26.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.514+0000 7f047c739640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:26.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.514+0000 7f047c739640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:26.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.514+0000 7f047c739640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:05:26.518 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.514+0000 7f047c739640 1 -- 172.21.15.66:0/3387145409 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04740627b0 con 0x7f04740b4d60 2024-03-25T18:05:26.829 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.827+0000 7f046affd640 1 -- 172.21.15.66:0/3387145409 <== mon.2 v1:172.21.15.66:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0464001c70 con 0x7f04740b4d60 2024-03-25T18:05:26.829 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.827+0000 7f046affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:26.829 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.827+0000 7f046affd640 1 -- 172.21.15.66:0/3387145409 <== mon.2 v1:172.21.15.66:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f0464003a60 con 0x7f04740b4d60 2024-03-25T18:05:26.829 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.827+0000 7f046affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:26.829 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.827+0000 7f046affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:05:26.829 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.827+0000 7f046affd640 10 monclient: dump: 2024-03-25T18:05:26.829 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:26.829 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:26.830 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:26.830 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:26.830 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:26.830 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:26.830 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:26.830 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:26.830 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:26.830 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:26.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.827+0000 7f0478cd2640 4 set_mon_vals no callback set 2024-03-25T18:05:26.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.829+0000 7f046affd640 1 -- 172.21.15.66:0/3387145409 <== mon.2 v1:172.21.15.66:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f0464003c40 con 0x7f04740b4d60 2024-03-25T18:05:26.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.829+0000 7f047a4d5640 10 monclient: get_auth_request con 0x7f045806e110 auth_method 0 2024-03-25T18:05:26.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.830+0000 7f046affd640 10 client.4292.objecter ms_handle_connect 0x7f045806e110 2024-03-25T18:05:26.836 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.834+0000 7f046affd640 1 -- 172.21.15.66:0/3387145409 <== mon.2 v1:172.21.15.66:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f0464054930 con 0x7f04740b4d60 2024-03-25T18:05:26.836 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.834+0000 7f046affd640 10 client.4292.objecter ms_dispatch 0x7f0474000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:05:26.836 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.834+0000 7f046affd640 3 client.4292.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:05:26.836 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.834+0000 7f046affd640 3 client.4292.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:05:26.836 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.834+0000 7f046affd640 20 client.4292.objecter dump_active .. 0 homeless 2024-03-25T18:05:26.836 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.834+0000 7f046affd640 1 -- 172.21.15.66:0/3387145409 <== mon.2 v1:172.21.15.66:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0464050440 con 0x7f04740b4d60 2024-03-25T18:05:26.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.834+0000 7f046affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:26.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:26.834+0000 7f046affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:05:27.092 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:27.090+0000 7f047c739640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:27.092 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:27.090+0000 7f047c739640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:27.092 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:27.090+0000 7f047c739640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:05:27.092 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:27.090+0000 7f047c739640 1 -- 172.21.15.66:0/3387145409 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0474069f40 con 0x7f04740b4d60 2024-03-25T18:05:27.133 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:27.132+0000 7f046affd640 1 -- 172.21.15.66:0/3387145409 <== mon.2 v1:172.21.15.66:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f0464052370 con 0x7f04740b4d60 2024-03-25T18:05:27.133 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:27.132+0000 7f046affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:27.133 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:27.132+0000 7f046affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:05:27.133 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:05:27.134 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:05:27.136 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:27.134+0000 7f047c739640 10 librados: watch_flush enter 2024-03-25T18:05:27.136 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:27.134+0000 7f047c739640 10 librados: watch_flush exit 2024-03-25T18:05:27.136 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:27.134+0000 7f047c739640 20 client.4292.objecter shutdown clearing up homeless session... 2024-03-25T18:05:27.136 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:27.134+0000 7f047c739640 10 client.4292.objecter successfully canceled tick 2024-03-25T18:05:27.136 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:27.134+0000 7f047c739640 1 -- 172.21.15.66:0/3387145409 >> v1:172.21.15.193:6800/2275676803 conn(0x7f045806e110 legacy=0x7f04580705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:27.136 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:27.134+0000 7f047c739640 10 monclient: shutdown 2024-03-25T18:05:27.136 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:27.134+0000 7f047c739640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:27.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:27.134+0000 7f047c739640 1 -- 172.21.15.66:0/3387145409 >> v1:172.21.15.66:6790/0 conn(0x7f04740b4d60 legacy=0x7f0474137130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:27.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:27.134+0000 7f047c739640 1 -- 172.21.15.66:0/3387145409 shutdown_connections 2024-03-25T18:05:27.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:27.134+0000 7f047c739640 1 -- 172.21.15.66:0/3387145409 >> 172.21.15.66:0/3387145409 conn(0x7f04740617a0 msgr2=0x7f04740b64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:27.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:27.134+0000 7f047c739640 1 -- 172.21.15.66:0/3387145409 shutdown_connections 2024-03-25T18:05:27.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:27.134+0000 7f047c739640 1 -- 172.21.15.66:0/3387145409 wait complete. 2024-03-25T18:05:27.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:27.134+0000 7f047c739640 1 librados: shutdown 2024-03-25T18:05:27.148 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:05:33.955 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:05:34.080 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.078+0000 7f0818363640 10 monclient: get_monmap_and_config 2024-03-25T18:05:34.080 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.078+0000 7f0818363640 10 monclient: build_initial_monmap 2024-03-25T18:05:34.080 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.078+0000 7f0818363640 10 monclient: monmap: 2024-03-25T18:05:34.080 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:05:34.080 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:34.080 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:05:34.080073+0000 2024-03-25T18:05:34.080 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:05:34.080073+0000 2024-03-25T18:05:34.080 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:34.080 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:34.080 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:05:34.080 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:05:34.080 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:05:34.081 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:34.081 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.079+0000 7f0818363640 1 Processor -- start 2024-03-25T18:05:34.081 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.079+0000 7f0818363640 1 -- start start 2024-03-25T18:05:34.081 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.079+0000 7f0818363640 10 monclient: init 2024-03-25T18:05:34.081 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.080+0000 7f0818363640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:34.082 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.080+0000 7f0818363640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:05:34.082 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.080+0000 7f0818363640 10 monclient(hunting): picked mon.noname-b con 0x7f08100a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:05:34.082 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.080+0000 7f0818363640 10 monclient(hunting): picked mon.noname-c con 0x7f081013f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:05:34.082 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.080+0000 7f0818363640 10 monclient(hunting): picked mon.noname-a con 0x7f08101430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:05:34.082 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.080+0000 7f0818363640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0810142520 con 0x7f08101430c0 2024-03-25T18:05:34.082 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.080+0000 7f0818363640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08100b28a0 con 0x7f081013f710 2024-03-25T18:05:34.082 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.080+0000 7f0818363640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08100b2a30 con 0x7f08100a98f0 2024-03-25T18:05:34.082 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.080+0000 7f0818363640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:34.082 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.080+0000 7f0818363640 10 monclient(hunting): authenticate will time out at 7083.140492s 2024-03-25T18:05:34.097 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.096+0000 7f08160ff640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f08100a98f0 0x7f08100a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:42752/0 (socket says 172.21.15.66:42752) 2024-03-25T18:05:34.097 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.096+0000 7f08160ff640 1 -- 172.21.15.66:0/392376716 learned_addr learned my addr 172.21.15.66:0/392376716 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:05:34.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.109+0000 7f08150fd640 1 -- 172.21.15.66:0/392376716 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2344295620 0 0) 0x7f08100b28a0 con 0x7f081013f710 2024-03-25T18:05:34.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.109+0000 7f08150fd640 10 monclient(hunting): _negotiate 2024-03-25T18:05:34.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.109+0000 7f08150fd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:34.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.109+0000 7f08150fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:34.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.109+0000 7f08150fd640 10 monclient(hunting): my global_id is 4294 2024-03-25T18:05:34.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.109+0000 7f08150fd640 1 -- 172.21.15.66:0/392376716 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07fc003520 con 0x7f081013f710 2024-03-25T18:05:34.111 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.110+0000 7f08150fd640 1 -- 172.21.15.66:0/392376716 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 861277122 0 0) 0x7f08100b2a30 con 0x7f08100a98f0 2024-03-25T18:05:34.111 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.110+0000 7f08150fd640 10 monclient(hunting): _negotiate 2024-03-25T18:05:34.111 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.110+0000 7f08150fd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:34.111 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.110+0000 7f08150fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:34.111 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.110+0000 7f08150fd640 10 monclient(hunting): my global_id is 4295 2024-03-25T18:05:34.112 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.110+0000 7f08150fd640 1 -- 172.21.15.66:0/392376716 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08100b28a0 con 0x7f08100a98f0 2024-03-25T18:05:34.122 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.121+0000 7f08150fd640 1 -- 172.21.15.66:0/392376716 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204418726 0 0) 0x7f0810142520 con 0x7f08101430c0 2024-03-25T18:05:34.122 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.121+0000 7f08150fd640 10 monclient(hunting): _negotiate 2024-03-25T18:05:34.122 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.121+0000 7f08150fd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:34.122 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.121+0000 7f08150fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:34.122 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.121+0000 7f08150fd640 10 monclient(hunting): my global_id is 4299 2024-03-25T18:05:34.122 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.121+0000 7f08150fd640 1 -- 172.21.15.66:0/392376716 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08100b2a30 con 0x7f08101430c0 2024-03-25T18:05:34.127 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.125+0000 7f08150fd640 1 -- 172.21.15.66:0/392376716 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3088606367 0 0) 0x7f07fc003520 con 0x7f081013f710 2024-03-25T18:05:34.127 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.125+0000 7f08150fd640 1 -- 172.21.15.66:0/392376716 >> v1:172.21.15.66:6790/0 conn(0x7f08100a98f0 legacy=0x7f08100a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:34.127 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.125+0000 7f08150fd640 1 -- 172.21.15.66:0/392376716 >> v1:172.21.15.66:6789/0 conn(0x7f08101430c0 legacy=0x7f081014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:34.127 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.125+0000 7f08150fd640 10 monclient: _finish_hunting 0 2024-03-25T18:05:34.127 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.125+0000 7f08150fd640 1 monclient: found mon.noname-c 2024-03-25T18:05:34.127 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.125+0000 7f08150fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:34.127 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.125+0000 7f08150fd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:05:34.127 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.125+0000 7f08150fd640 1 -- 172.21.15.66:0/392376716 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08100b44c0 con 0x7f081013f710 2024-03-25T18:05:34.127 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.125+0000 7f08150fd640 10 monclient: _finish_auth 0 2024-03-25T18:05:34.127 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.125+0000 7f08150fd640 10 monclient: _check_auth_tickets 2024-03-25T18:05:34.127 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.125+0000 7f08150fd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:34.127 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.125+0000 7f08150fd640 1 -- 172.21.15.66:0/392376716 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f0800001690 con 0x7f081013f710 2024-03-25T18:05:34.128 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.125+0000 7f08150fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:34.128 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.125+0000 7f08150fd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:05:34.128 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.125+0000 7f08150fd640 10 monclient: dump: 2024-03-25T18:05:34.128 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:34.128 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:34.128 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:34.128 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:34.128 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:34.128 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:34.128 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:34.128 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:34.128 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:34.128 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:34.128 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.125+0000 7f0818363640 5 monclient: authenticate success, global_id 4294 2024-03-25T18:05:34.128 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.125+0000 7f0818363640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:34.142 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.141+0000 7f08150fd640 1 -- 172.21.15.66:0/392376716 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08000017f0 con 0x7f081013f710 2024-03-25T18:05:34.143 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.141+0000 7f08150fd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:34.143 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.141+0000 7f08150fd640 1 -- 172.21.15.66:0/392376716 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f0800001c00 con 0x7f081013f710 2024-03-25T18:05:34.143 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.141+0000 7f08150fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:34.143 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.141+0000 7f08150fd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:34.143 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.141+0000 7f08150fd640 10 monclient: dump: 2024-03-25T18:05:34.143 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:34.143 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:34.143 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:34.143 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:34.143 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:34.143 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:34.143 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:34.143 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:34.143 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:34.143 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:34.143 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.141+0000 7f0818363640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:34.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.141+0000 7f0818363640 4 set_mon_vals no callback set 2024-03-25T18:05:34.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.142+0000 7f0818363640 10 monclient: shutdown 2024-03-25T18:05:34.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.142+0000 7f0818363640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:34.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.142+0000 7f0818363640 1 -- 172.21.15.66:0/392376716 >> v1:172.21.15.193:6789/0 conn(0x7f081013f710 legacy=0x7f08101429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:34.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.142+0000 7f0818363640 1 -- 172.21.15.66:0/392376716 shutdown_connections 2024-03-25T18:05:34.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.142+0000 7f0818363640 1 -- 172.21.15.66:0/392376716 >> 172.21.15.66:0/392376716 conn(0x7f08100b7ea0 msgr2=0x7f08100b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:34.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.142+0000 7f0818363640 1 -- 172.21.15.66:0/392376716 shutdown_connections 2024-03-25T18:05:34.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.142+0000 7f0818363640 1 -- 172.21.15.66:0/392376716 wait complete. 2024-03-25T18:05:34.146 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.143+0000 7f0818363640 10 monclient: build_initial_monmap 2024-03-25T18:05:34.146 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.143+0000 7f0818363640 10 monclient: monmap: 2024-03-25T18:05:34.146 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:05:34.146 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:34.146 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:05:34.146 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:05:34.146 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:34.146 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:05:34.146 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:05:34.146 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:05:34.146 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:05:34.146 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:34.146 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.143+0000 7f0818363640 1 librados: starting msgr at 2024-03-25T18:05:34.146 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.143+0000 7f0818363640 1 librados: starting objecter 2024-03-25T18:05:34.146 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.144+0000 7f0818363640 1 Processor -- start 2024-03-25T18:05:34.146 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.144+0000 7f0818363640 1 -- start start 2024-03-25T18:05:34.147 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.144+0000 7f0818363640 1 librados: setting wanted keys 2024-03-25T18:05:34.147 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.144+0000 7f0818363640 1 librados: calling monclient init 2024-03-25T18:05:34.147 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.144+0000 7f0818363640 10 monclient: init 2024-03-25T18:05:34.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.145+0000 7f0818363640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:34.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.145+0000 7f0818363640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:05:34.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.145+0000 7f0818363640 10 monclient(hunting): picked mon.noname-c con 0x7f08100a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:05:34.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.145+0000 7f0818363640 10 monclient(hunting): picked mon.noname-a con 0x7f081013f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:05:34.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.145+0000 7f0818363640 10 monclient(hunting): picked mon.noname-b con 0x7f08101430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:05:34.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.145+0000 7f0818363640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0810138250 con 0x7f081013f710 2024-03-25T18:05:34.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.145+0000 7f0818363640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0810137d50 con 0x7f08101430c0 2024-03-25T18:05:34.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.145+0000 7f0818363640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0810137ee0 con 0x7f08100a98f0 2024-03-25T18:05:34.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.145+0000 7f0818363640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:34.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.145+0000 7f0818363640 10 monclient(hunting): authenticate will time out at 7083.205467s 2024-03-25T18:05:34.151 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.150+0000 7f08160ff640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f08100a98f0 0x7f0810137640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:42800/0 (socket says 172.21.15.66:42800) 2024-03-25T18:05:34.151 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.150+0000 7f08160ff640 1 -- 172.21.15.66:0/3539467694 learned_addr learned my addr 172.21.15.66:0/3539467694 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:05:34.152 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.151+0000 7f0806ffd640 10 client.?.objecter ms_handle_connect 0x7f08100a98f0 2024-03-25T18:05:34.152 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.151+0000 7f0806ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:34.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.155+0000 7f0806ffd640 10 client.?.objecter ms_handle_connect 0x7f08101430c0 2024-03-25T18:05:34.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.155+0000 7f0806ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:34.160 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.159+0000 7f0806ffd640 10 client.?.objecter ms_handle_connect 0x7f081013f710 2024-03-25T18:05:34.160 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.159+0000 7f0806ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:34.160 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.159+0000 7f0806ffd640 1 -- 172.21.15.66:0/3539467694 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34792225 0 0) 0x7f0810137ee0 con 0x7f08100a98f0 2024-03-25T18:05:34.160 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.159+0000 7f0806ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:05:34.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.159+0000 7f0806ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:34.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.159+0000 7f0806ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:34.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.159+0000 7f0806ffd640 10 monclient(hunting): my global_id is 4298 2024-03-25T18:05:34.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.159+0000 7f0806ffd640 1 -- 172.21.15.66:0/3539467694 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07f4004eb0 con 0x7f08100a98f0 2024-03-25T18:05:34.172 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.170+0000 7f0806ffd640 1 -- 172.21.15.66:0/3539467694 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2736950441 0 0) 0x7f0810137d50 con 0x7f08101430c0 2024-03-25T18:05:34.172 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.170+0000 7f0806ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:05:34.172 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.170+0000 7f0806ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:34.172 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.170+0000 7f0806ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:34.172 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.170+0000 7f0806ffd640 10 monclient(hunting): my global_id is 4297 2024-03-25T18:05:34.172 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.170+0000 7f0806ffd640 1 -- 172.21.15.66:0/3539467694 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0810137ee0 con 0x7f08101430c0 2024-03-25T18:05:34.172 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.170+0000 7f0806ffd640 1 -- 172.21.15.66:0/3539467694 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4053949005 0 0) 0x7f0810138250 con 0x7f081013f710 2024-03-25T18:05:34.172 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.170+0000 7f0806ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:05:34.172 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.170+0000 7f0806ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:34.172 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.170+0000 7f0806ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:34.172 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.170+0000 7f0806ffd640 10 monclient(hunting): my global_id is 4302 2024-03-25T18:05:34.172 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.170+0000 7f0806ffd640 1 -- 172.21.15.66:0/3539467694 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0810137d50 con 0x7f081013f710 2024-03-25T18:05:34.179 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.177+0000 7f0806ffd640 1 -- 172.21.15.66:0/3539467694 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3190875080 0 0) 0x7f07f4004eb0 con 0x7f08100a98f0 2024-03-25T18:05:34.179 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.177+0000 7f0806ffd640 1 -- 172.21.15.66:0/3539467694 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0810138250 con 0x7f08100a98f0 2024-03-25T18:05:34.179 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.177+0000 7f0806ffd640 1 -- 172.21.15.66:0/3539467694 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f080c001d70 con 0x7f08100a98f0 2024-03-25T18:05:34.179 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.177+0000 7f0806ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:34.179 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.177+0000 7f0806ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:05:34.179 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.177+0000 7f0806ffd640 10 monclient(hunting): dump: 2024-03-25T18:05:34.179 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:34.179 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:34.179 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:34.179 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:34.179 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:34.180 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:34.180 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:34.180 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:34.180 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:34.180 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:34.180 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.177+0000 7f0806ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:34.189 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.187+0000 7f0806ffd640 1 -- 172.21.15.66:0/3539467694 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4051206330 0 0) 0x7f0810137ee0 con 0x7f08101430c0 2024-03-25T18:05:34.189 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.188+0000 7f0806ffd640 1 -- 172.21.15.66:0/3539467694 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07f4004eb0 con 0x7f08101430c0 2024-03-25T18:05:34.189 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.188+0000 7f0806ffd640 1 -- 172.21.15.66:0/3539467694 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f08080017d0 con 0x7f08101430c0 2024-03-25T18:05:34.189 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.188+0000 7f0806ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:34.189 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.188+0000 7f0806ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:34.189 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.188+0000 7f0806ffd640 10 monclient(hunting): dump: 2024-03-25T18:05:34.189 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:34.189 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:34.189 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:34.189 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:34.190 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:34.190 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:34.190 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:34.190 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:34.190 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:34.190 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:34.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.190+0000 7f0806ffd640 1 -- 172.21.15.66:0/3539467694 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2894767131 0 0) 0x7f0810137d50 con 0x7f081013f710 2024-03-25T18:05:34.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.190+0000 7f0806ffd640 1 -- 172.21.15.66:0/3539467694 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0810137ee0 con 0x7f081013f710 2024-03-25T18:05:34.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.190+0000 7f0806ffd640 1 -- 172.21.15.66:0/3539467694 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f0800001b00 con 0x7f081013f710 2024-03-25T18:05:34.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.190+0000 7f0806ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:34.192 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.190+0000 7f0806ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:34.192 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.190+0000 7f0806ffd640 10 monclient(hunting): dump: 2024-03-25T18:05:34.192 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:34.192 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:34.192 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:34.192 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:34.192 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:34.192 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:34.192 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:34.192 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:34.192 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:34.192 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:34.198 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.196+0000 7f0806ffd640 1 -- 172.21.15.66:0/3539467694 <== mon.2 v1:172.21.15.66:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3127518659 0 0) 0x7f0810138250 con 0x7f08100a98f0 2024-03-25T18:05:34.198 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.196+0000 7f0806ffd640 1 -- 172.21.15.66:0/3539467694 >> v1:172.21.15.193:6789/0 conn(0x7f08101430c0 legacy=0x7f08101d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:34.198 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.196+0000 7f0806ffd640 1 -- 172.21.15.66:0/3539467694 >> v1:172.21.15.66:6789/0 conn(0x7f081013f710 legacy=0x7f08101d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:34.198 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.196+0000 7f0806ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:05:34.198 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.196+0000 7f0806ffd640 1 monclient: found mon.c 2024-03-25T18:05:34.198 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.196+0000 7f0806ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:34.198 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.196+0000 7f0806ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:05:34.198 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.196+0000 7f0806ffd640 1 -- 172.21.15.66:0/3539467694 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08100b7730 con 0x7f08100a98f0 2024-03-25T18:05:34.198 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.196+0000 7f0806ffd640 10 monclient: _finish_auth 0 2024-03-25T18:05:34.198 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.196+0000 7f0806ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:05:34.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.196+0000 7f0806ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:34.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.196+0000 7f0818363640 5 monclient: authenticate success, global_id 4298 2024-03-25T18:05:34.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.196+0000 7f0818363640 10 monclient: _renew_subs 2024-03-25T18:05:34.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.196+0000 7f0818363640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:05:34.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.196+0000 7f0818363640 1 -- 172.21.15.66:0/3539467694 --> v1:172.21.15.66:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08100b7920 con 0x7f08100a98f0 2024-03-25T18:05:34.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.196+0000 7f0818363640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:34.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.196+0000 7f0818363640 10 monclient: _renew_subs 2024-03-25T18:05:34.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.196+0000 7f0818363640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:05:34.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.196+0000 7f0818363640 1 -- 172.21.15.66:0/3539467694 --> v1:172.21.15.66:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08101e33d0 con 0x7f08100a98f0 2024-03-25T18:05:34.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.196+0000 7f0818363640 1 librados: init done 2024-03-25T18:05:34.200 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.197+0000 7f0818363640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:34.200 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.197+0000 7f0818363640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:34.200 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.197+0000 7f0818363640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:05:34.200 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.197+0000 7f0818363640 1 -- 172.21.15.66:0/3539467694 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0810139710 con 0x7f08100a98f0 2024-03-25T18:05:34.514 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.513+0000 7f0806ffd640 1 -- 172.21.15.66:0/3539467694 <== mon.2 v1:172.21.15.66:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f080c004b60 con 0x7f08100a98f0 2024-03-25T18:05:34.515 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.513+0000 7f0806ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:34.515 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.513+0000 7f0806ffd640 1 -- 172.21.15.66:0/3539467694 <== mon.2 v1:172.21.15.66:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f080c004e70 con 0x7f08100a98f0 2024-03-25T18:05:34.515 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.513+0000 7f0806ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:34.515 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.513+0000 7f0806ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:05:34.515 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.513+0000 7f0806ffd640 10 monclient: dump: 2024-03-25T18:05:34.515 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:34.515 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:34.515 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:34.515 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:34.515 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:34.515 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:34.516 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:34.516 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:34.516 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:34.516 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:34.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.513+0000 7f08148fc640 4 set_mon_vals no callback set 2024-03-25T18:05:34.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.514+0000 7f0806ffd640 1 -- 172.21.15.66:0/3539467694 <== mon.2 v1:172.21.15.66:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f080c01aec0 con 0x7f08100a98f0 2024-03-25T18:05:34.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.514+0000 7f08158fe640 10 monclient: get_auth_request con 0x7f07f406e110 auth_method 0 2024-03-25T18:05:34.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.515+0000 7f0806ffd640 10 client.4298.objecter ms_handle_connect 0x7f07f406e110 2024-03-25T18:05:34.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.519+0000 7f0806ffd640 1 -- 172.21.15.66:0/3539467694 <== mon.2 v1:172.21.15.66:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f080c054c20 con 0x7f08100a98f0 2024-03-25T18:05:34.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.519+0000 7f0806ffd640 10 client.4298.objecter ms_dispatch 0x7f0810000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:05:34.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.519+0000 7f0806ffd640 3 client.4298.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:05:34.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.519+0000 7f0806ffd640 3 client.4298.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:05:34.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.519+0000 7f0806ffd640 20 client.4298.objecter dump_active .. 0 homeless 2024-03-25T18:05:34.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.519+0000 7f0806ffd640 1 -- 172.21.15.66:0/3539467694 <== mon.2 v1:172.21.15.66:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f080c050560 con 0x7f08100a98f0 2024-03-25T18:05:34.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.519+0000 7f0806ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:34.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.519+0000 7f0806ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:05:34.780 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.778+0000 7f0818363640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:34.780 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.778+0000 7f0818363640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:34.780 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.778+0000 7f0818363640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:05:34.780 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.778+0000 7f0818363640 1 -- 172.21.15.66:0/3539467694 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08100627b0 con 0x7f08100a98f0 2024-03-25T18:05:34.846 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.845+0000 7f0806ffd640 1 -- 172.21.15.66:0/3539467694 <== mon.2 v1:172.21.15.66:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f080c052660 con 0x7f08100a98f0 2024-03-25T18:05:34.846 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.845+0000 7f0806ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:34.847 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.845+0000 7f0806ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:05:34.847 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:05:34.847 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:05:34.849 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.847+0000 7f0818363640 10 librados: watch_flush enter 2024-03-25T18:05:34.850 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.847+0000 7f0818363640 10 librados: watch_flush exit 2024-03-25T18:05:34.850 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.847+0000 7f0818363640 20 client.4298.objecter shutdown clearing up homeless session... 2024-03-25T18:05:34.850 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.847+0000 7f0818363640 10 client.4298.objecter successfully canceled tick 2024-03-25T18:05:34.850 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.848+0000 7f0818363640 1 -- 172.21.15.66:0/3539467694 >> v1:172.21.15.193:6800/2275676803 conn(0x7f07f406e110 legacy=0x7f07f40705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:34.850 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.848+0000 7f0818363640 10 monclient: shutdown 2024-03-25T18:05:34.850 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.848+0000 7f0818363640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:34.850 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.848+0000 7f0818363640 1 -- 172.21.15.66:0/3539467694 >> v1:172.21.15.66:6790/0 conn(0x7f08100a98f0 legacy=0x7f0810137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:34.850 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.848+0000 7f0818363640 1 -- 172.21.15.66:0/3539467694 shutdown_connections 2024-03-25T18:05:34.850 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.848+0000 7f0818363640 1 -- 172.21.15.66:0/3539467694 >> 172.21.15.66:0/3539467694 conn(0x7f08100b7ea0 msgr2=0x7f0810074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:34.850 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.848+0000 7f0818363640 1 -- 172.21.15.66:0/3539467694 shutdown_connections 2024-03-25T18:05:34.850 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.848+0000 7f0818363640 1 -- 172.21.15.66:0/3539467694 wait complete. 2024-03-25T18:05:34.850 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:34.848+0000 7f0818363640 1 librados: shutdown 2024-03-25T18:05:34.860 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:05:41.668 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:05:41.786 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.786+0000 7fefebe93640 10 monclient: get_monmap_and_config 2024-03-25T18:05:41.787 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.786+0000 7fefebe93640 10 monclient: build_initial_monmap 2024-03-25T18:05:41.787 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.786+0000 7fefebe93640 10 monclient: monmap: 2024-03-25T18:05:41.787 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:05:41.787 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:41.787 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:05:41.786598+0000 2024-03-25T18:05:41.787 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:05:41.786598+0000 2024-03-25T18:05:41.787 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:41.787 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:41.787 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:05:41.787 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:05:41.788 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:05:41.788 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:41.788 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.786+0000 7fefebe93640 1 Processor -- start 2024-03-25T18:05:41.789 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.787+0000 7fefebe93640 1 -- start start 2024-03-25T18:05:41.789 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.787+0000 7fefebe93640 10 monclient: init 2024-03-25T18:05:41.789 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.787+0000 7fefebe93640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:41.789 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.787+0000 7fefebe93640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:05:41.789 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.787+0000 7fefebe93640 10 monclient(hunting): picked mon.noname-a con 0x7fefe40a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:05:41.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.787+0000 7fefebe93640 10 monclient(hunting): picked mon.noname-c con 0x7fefe413f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:05:41.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.788+0000 7fefebe93640 10 monclient(hunting): picked mon.noname-b con 0x7fefe41430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:05:41.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.788+0000 7fefebe93640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe4142520 con 0x7fefe40a98f0 2024-03-25T18:05:41.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.788+0000 7fefebe93640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe40b28a0 con 0x7fefe413f710 2024-03-25T18:05:41.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.788+0000 7fefebe93640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe40b2a30 con 0x7fefe41430c0 2024-03-25T18:05:41.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.788+0000 7fefebe93640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:41.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.788+0000 7fefebe93640 10 monclient(hunting): authenticate will time out at 7090.847209s 2024-03-25T18:05:41.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.790+0000 7fefe9c2f640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7fefe40a98f0 0x7fefe40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:44242/0 (socket says 172.21.15.66:44242) 2024-03-25T18:05:41.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.790+0000 7fefe9c2f640 1 -- 172.21.15.66:0/2781570594 learned_addr learned my addr 172.21.15.66:0/2781570594 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:05:41.829 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.799+0000 7fefe8c2d640 1 -- 172.21.15.66:0/2781570594 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2473513136 0 0) 0x7fefe4142520 con 0x7fefe40a98f0 2024-03-25T18:05:41.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.799+0000 7fefe8c2d640 10 monclient(hunting): _negotiate 2024-03-25T18:05:41.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.799+0000 7fefe8c2d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:41.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.799+0000 7fefe8c2d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:41.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.799+0000 7fefe8c2d640 10 monclient(hunting): my global_id is 4305 2024-03-25T18:05:41.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.799+0000 7fefe8c2d640 1 -- 172.21.15.66:0/2781570594 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefd0003520 con 0x7fefe40a98f0 2024-03-25T18:05:41.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.816+0000 7fefe8c2d640 1 -- 172.21.15.66:0/2781570594 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 567121788 0 0) 0x7fefd0003520 con 0x7fefe40a98f0 2024-03-25T18:05:41.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.816+0000 7fefe8c2d640 1 -- 172.21.15.66:0/2781570594 >> v1:172.21.15.66:6790/0 conn(0x7fefe41430c0 legacy=0x7fefe414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:41.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.816+0000 7fefe8c2d640 1 -- 172.21.15.66:0/2781570594 >> v1:172.21.15.193:6789/0 conn(0x7fefe413f710 legacy=0x7fefe41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:41.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.816+0000 7fefe8c2d640 10 monclient: _finish_hunting 0 2024-03-25T18:05:41.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.816+0000 7fefe8c2d640 1 monclient: found mon.noname-a 2024-03-25T18:05:41.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.816+0000 7fefe8c2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:41.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.816+0000 7fefe8c2d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.66:6789/0 2024-03-25T18:05:41.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.816+0000 7fefe8c2d640 1 -- 172.21.15.66:0/2781570594 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe40b44c0 con 0x7fefe40a98f0 2024-03-25T18:05:41.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.816+0000 7fefe8c2d640 10 monclient: _finish_auth 0 2024-03-25T18:05:41.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.816+0000 7fefe8c2d640 10 monclient: _check_auth_tickets 2024-03-25T18:05:41.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.816+0000 7fefe8c2d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:41.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.816+0000 7fefe8c2d640 1 -- 172.21.15.66:0/2781570594 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fefe0001690 con 0x7fefe40a98f0 2024-03-25T18:05:41.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.816+0000 7fefebe93640 5 monclient: authenticate success, global_id 4305 2024-03-25T18:05:41.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.816+0000 7fefebe93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:41.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.816+0000 7fefe8c2d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:41.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.816+0000 7fefe8c2d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:05:41.832 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.816+0000 7fefe8c2d640 10 monclient: dump: 2024-03-25T18:05:41.832 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:41.832 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:41.832 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:41.832 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:41.832 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:41.832 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:41.832 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:41.832 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:41.832 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:41.832 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:41.833 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.816+0000 7fefebe93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:41.835 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.830+0000 7fefe8c2d640 1 -- 172.21.15.66:0/2781570594 <== mon.0 v1:172.21.15.66:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefe00017f0 con 0x7fefe40a98f0 2024-03-25T18:05:41.835 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.830+0000 7fefe8c2d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:41.835 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.830+0000 7fefe8c2d640 1 -- 172.21.15.66:0/2781570594 <== mon.0 v1:172.21.15.66:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fefe0001c00 con 0x7fefe40a98f0 2024-03-25T18:05:41.836 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.830+0000 7fefe8c2d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:41.836 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.830+0000 7fefe8c2d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:41.836 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.830+0000 7fefe8c2d640 10 monclient: dump: 2024-03-25T18:05:41.836 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:41.836 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:41.836 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:41.836 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:41.836 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:41.836 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:41.836 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:41.836 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:41.837 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:41.837 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:41.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.830+0000 7fefebe93640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:41.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.830+0000 7fefebe93640 4 set_mon_vals no callback set 2024-03-25T18:05:41.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.831+0000 7fefebe93640 10 monclient: shutdown 2024-03-25T18:05:41.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.831+0000 7fefebe93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:41.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.831+0000 7fefebe93640 1 -- 172.21.15.66:0/2781570594 >> v1:172.21.15.66:6789/0 conn(0x7fefe40a98f0 legacy=0x7fefe40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:41.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.831+0000 7fefebe93640 1 -- 172.21.15.66:0/2781570594 shutdown_connections 2024-03-25T18:05:41.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.831+0000 7fefebe93640 1 -- 172.21.15.66:0/2781570594 >> 172.21.15.66:0/2781570594 conn(0x7fefe40b7ea0 msgr2=0x7fefe40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:41.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.831+0000 7fefebe93640 1 -- 172.21.15.66:0/2781570594 shutdown_connections 2024-03-25T18:05:41.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.831+0000 7fefebe93640 1 -- 172.21.15.66:0/2781570594 wait complete. 2024-03-25T18:05:41.838 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.832+0000 7fefebe93640 10 monclient: build_initial_monmap 2024-03-25T18:05:41.838 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.832+0000 7fefebe93640 10 monclient: monmap: 2024-03-25T18:05:41.838 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:05:41.838 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:41.838 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:05:41.838 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:05:41.838 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:41.838 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:05:41.838 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:05:41.838 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:05:41.838 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:05:41.838 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:41.839 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.832+0000 7fefebe93640 1 librados: starting msgr at 2024-03-25T18:05:41.839 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.832+0000 7fefebe93640 1 librados: starting objecter 2024-03-25T18:05:41.839 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.832+0000 7fefebe93640 1 Processor -- start 2024-03-25T18:05:41.839 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.832+0000 7fefebe93640 1 -- start start 2024-03-25T18:05:41.839 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.832+0000 7fefebe93640 1 librados: setting wanted keys 2024-03-25T18:05:41.839 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.832+0000 7fefebe93640 1 librados: calling monclient init 2024-03-25T18:05:41.839 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.832+0000 7fefebe93640 10 monclient: init 2024-03-25T18:05:41.839 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.833+0000 7fefebe93640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:41.839 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.833+0000 7fefebe93640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:05:41.839 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.833+0000 7fefebe93640 10 monclient(hunting): picked mon.noname-b con 0x7fefe40a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:05:41.839 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.833+0000 7fefebe93640 10 monclient(hunting): picked mon.noname-c con 0x7fefe413f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:05:41.839 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.833+0000 7fefebe93640 10 monclient(hunting): picked mon.noname-a con 0x7fefe41430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:05:41.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.833+0000 7fefebe93640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe4135ac0 con 0x7fefe41430c0 2024-03-25T18:05:41.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.833+0000 7fefebe93640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe4137030 con 0x7fefe40a98f0 2024-03-25T18:05:41.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.833+0000 7fefebe93640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe41372a0 con 0x7fefe413f710 2024-03-25T18:05:41.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.833+0000 7fefebe93640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:41.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.833+0000 7fefebe93640 10 monclient(hunting): authenticate will time out at 7090.892661s 2024-03-25T18:05:41.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.844+0000 7fefea430640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7fefe41430c0 0x7fefe41e26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:44296/0 (socket says 172.21.15.66:44296) 2024-03-25T18:05:41.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.844+0000 7fefea430640 1 -- 172.21.15.66:0/3432457768 learned_addr learned my addr 172.21.15.66:0/3432457768 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:05:41.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.845+0000 7fefda7fc640 10 client.?.objecter ms_handle_connect 0x7fefe41430c0 2024-03-25T18:05:41.846 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.845+0000 7fefda7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:41.849 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.849+0000 7fefda7fc640 10 client.?.objecter ms_handle_connect 0x7fefe40a98f0 2024-03-25T18:05:41.849 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.849+0000 7fefda7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:41.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.856+0000 7fefda7fc640 1 -- 172.21.15.66:0/3432457768 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473502706 0 0) 0x7fefe4135ac0 con 0x7fefe41430c0 2024-03-25T18:05:41.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.856+0000 7fefda7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:41.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.856+0000 7fefda7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:41.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.856+0000 7fefda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:41.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.856+0000 7fefda7fc640 10 monclient(hunting): my global_id is 4308 2024-03-25T18:05:41.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.856+0000 7fefda7fc640 1 -- 172.21.15.66:0/3432457768 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefc8004eb0 con 0x7fefe41430c0 2024-03-25T18:05:41.860 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.860+0000 7fefda7fc640 1 -- 172.21.15.66:0/3432457768 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2440507428 0 0) 0x7fefe4137030 con 0x7fefe40a98f0 2024-03-25T18:05:41.860 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.860+0000 7fefda7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:41.860 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.860+0000 7fefda7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:41.860 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.860+0000 7fefda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:41.860 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.860+0000 7fefda7fc640 10 monclient(hunting): my global_id is 4303 2024-03-25T18:05:41.860 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.860+0000 7fefda7fc640 1 -- 172.21.15.66:0/3432457768 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefe4135ac0 con 0x7fefe40a98f0 2024-03-25T18:05:41.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.867+0000 7fefda7fc640 10 client.?.objecter ms_handle_connect 0x7fefe413f710 2024-03-25T18:05:41.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.867+0000 7fefda7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:41.881 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.880+0000 7fefda7fc640 1 -- 172.21.15.66:0/3432457768 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 305593946 0 0) 0x7fefc8004eb0 con 0x7fefe41430c0 2024-03-25T18:05:41.881 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.880+0000 7fefda7fc640 1 -- 172.21.15.66:0/3432457768 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefe4137030 con 0x7fefe41430c0 2024-03-25T18:05:41.881 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.880+0000 7fefda7fc640 1 -- 172.21.15.66:0/3432457768 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fefdc001c90 con 0x7fefe41430c0 2024-03-25T18:05:41.881 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.880+0000 7fefda7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:41.881 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.880+0000 7fefda7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:05:41.881 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.880+0000 7fefda7fc640 10 monclient(hunting): dump: 2024-03-25T18:05:41.881 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:41.881 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:41.881 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:41.882 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:41.882 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:41.882 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:41.882 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:41.882 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:41.882 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:41.882 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:41.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.880+0000 7fefda7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:41.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.880+0000 7fefda7fc640 1 -- 172.21.15.66:0/3432457768 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 59806548 0 0) 0x7fefe4135ac0 con 0x7fefe40a98f0 2024-03-25T18:05:41.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.880+0000 7fefda7fc640 1 -- 172.21.15.66:0/3432457768 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefc8004eb0 con 0x7fefe40a98f0 2024-03-25T18:05:41.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.880+0000 7fefda7fc640 1 -- 172.21.15.66:0/3432457768 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fefe0001b00 con 0x7fefe40a98f0 2024-03-25T18:05:41.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.881+0000 7fefda7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:41.883 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.881+0000 7fefda7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:41.883 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.881+0000 7fefda7fc640 10 monclient(hunting): dump: 2024-03-25T18:05:41.883 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:41.883 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:41.883 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:41.883 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:41.883 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:41.883 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:41.883 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:41.883 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:41.883 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:41.883 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:41.884 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.881+0000 7fefda7fc640 1 -- 172.21.15.66:0/3432457768 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938254636 0 0) 0x7fefe41372a0 con 0x7fefe413f710 2024-03-25T18:05:41.884 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.881+0000 7fefda7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:05:41.884 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.881+0000 7fefda7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:41.884 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.881+0000 7fefda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:41.884 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.881+0000 7fefda7fc640 10 monclient(hunting): my global_id is 4304 2024-03-25T18:05:41.884 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.881+0000 7fefda7fc640 1 -- 172.21.15.66:0/3432457768 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefe4135ac0 con 0x7fefe413f710 2024-03-25T18:05:41.891 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.890+0000 7fefda7fc640 1 -- 172.21.15.66:0/3432457768 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1758846427 0 0) 0x7fefc8004eb0 con 0x7fefe40a98f0 2024-03-25T18:05:41.891 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.890+0000 7fefda7fc640 1 -- 172.21.15.66:0/3432457768 >> v1:172.21.15.66:6790/0 conn(0x7fefe413f710 legacy=0x7fefe41379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:41.891 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.890+0000 7fefda7fc640 1 -- 172.21.15.66:0/3432457768 >> v1:172.21.15.66:6789/0 conn(0x7fefe41430c0 legacy=0x7fefe41e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:41.891 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.890+0000 7fefda7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:05:41.891 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.890+0000 7fefda7fc640 1 monclient: found mon.b 2024-03-25T18:05:41.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.890+0000 7fefda7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:41.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.890+0000 7fefda7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:05:41.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.890+0000 7fefda7fc640 1 -- 172.21.15.66:0/3432457768 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe4137580 con 0x7fefe40a98f0 2024-03-25T18:05:41.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.890+0000 7fefda7fc640 10 monclient: _finish_auth 0 2024-03-25T18:05:41.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.890+0000 7fefda7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:05:41.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.890+0000 7fefda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:41.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.890+0000 7fefebe93640 5 monclient: authenticate success, global_id 4303 2024-03-25T18:05:41.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.890+0000 7fefebe93640 10 monclient: _renew_subs 2024-03-25T18:05:41.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.890+0000 7fefebe93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:05:41.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.890+0000 7fefebe93640 1 -- 172.21.15.66:0/3432457768 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefe41363b0 con 0x7fefe40a98f0 2024-03-25T18:05:41.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.890+0000 7fefebe93640 10 client.4303.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:41.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.890+0000 7fefebe93640 10 monclient: _renew_subs 2024-03-25T18:05:41.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.890+0000 7fefebe93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:05:41.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.890+0000 7fefebe93640 1 -- 172.21.15.66:0/3432457768 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fefe41353a0 con 0x7fefe40a98f0 2024-03-25T18:05:41.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.890+0000 7fefebe93640 1 librados: init done 2024-03-25T18:05:41.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.891+0000 7fefebe93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:41.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.891+0000 7fefebe93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:41.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.891+0000 7fefebe93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:05:41.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:41.891+0000 7fefebe93640 1 -- 172.21.15.66:0/3432457768 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefe40627b0 con 0x7fefe40a98f0 2024-03-25T18:05:42.172 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.172+0000 7fefda7fc640 1 -- 172.21.15.66:0/3432457768 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefe0004a90 con 0x7fefe40a98f0 2024-03-25T18:05:42.172 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.172+0000 7fefda7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:42.172 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.172+0000 7fefda7fc640 1 -- 172.21.15.66:0/3432457768 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fefe0004da0 con 0x7fefe40a98f0 2024-03-25T18:05:42.173 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.172+0000 7fefda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:42.173 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.172+0000 7fefda7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:42.173 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.172+0000 7fefda7fc640 10 monclient: dump: 2024-03-25T18:05:42.173 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:42.173 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:42.173 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:42.173 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:42.173 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:42.174 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:42.174 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:42.174 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:42.174 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:42.174 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:42.174 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.172+0000 7fefdbfff640 4 set_mon_vals no callback set 2024-03-25T18:05:42.175 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.173+0000 7fefda7fc640 1 -- 172.21.15.66:0/3432457768 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7fefe001ad70 con 0x7fefe40a98f0 2024-03-25T18:05:42.175 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.173+0000 7fefe942e640 10 monclient: get_auth_request con 0x7fefc806df50 auth_method 0 2024-03-25T18:05:42.175 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.174+0000 7fefda7fc640 10 client.4303.objecter ms_handle_connect 0x7fefc806df50 2024-03-25T18:05:42.179 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.178+0000 7fefda7fc640 1 -- 172.21.15.66:0/3432457768 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7fefe0054830 con 0x7fefe40a98f0 2024-03-25T18:05:42.179 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.178+0000 7fefda7fc640 10 client.4303.objecter ms_dispatch 0x7fefe4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:05:42.179 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.178+0000 7fefda7fc640 3 client.4303.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:05:42.179 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.178+0000 7fefda7fc640 3 client.4303.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:05:42.179 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.178+0000 7fefda7fc640 20 client.4303.objecter dump_active .. 0 homeless 2024-03-25T18:05:42.179 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.179+0000 7fefda7fc640 1 -- 172.21.15.66:0/3432457768 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fefe0050340 con 0x7fefe40a98f0 2024-03-25T18:05:42.179 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.179+0000 7fefda7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:42.179 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.179+0000 7fefda7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:05:42.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.452+0000 7fefebe93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:42.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.452+0000 7fefebe93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:42.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.452+0000 7fefebe93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:05:42.453 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.452+0000 7fefebe93640 1 -- 172.21.15.66:0/3432457768 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fefe4074cc0 con 0x7fefe40a98f0 2024-03-25T18:05:42.482 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.481+0000 7fefda7fc640 1 -- 172.21.15.66:0/3432457768 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7fefe0052270 con 0x7fefe40a98f0 2024-03-25T18:05:42.482 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.481+0000 7fefda7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:42.482 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.481+0000 7fefda7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:05:42.482 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:05:42.482 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:05:42.485 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.483+0000 7fefebe93640 10 librados: watch_flush enter 2024-03-25T18:05:42.485 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.483+0000 7fefebe93640 10 librados: watch_flush exit 2024-03-25T18:05:42.485 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.484+0000 7fefebe93640 20 client.4303.objecter shutdown clearing up homeless session... 2024-03-25T18:05:42.485 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.484+0000 7fefebe93640 10 client.4303.objecter successfully canceled tick 2024-03-25T18:05:42.485 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.484+0000 7fefebe93640 1 -- 172.21.15.66:0/3432457768 >> v1:172.21.15.193:6800/2275676803 conn(0x7fefc806df50 legacy=0x7fefc8070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:42.485 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.484+0000 7fefebe93640 10 monclient: shutdown 2024-03-25T18:05:42.485 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.484+0000 7fefebe93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:42.485 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.484+0000 7fefebe93640 1 -- 172.21.15.66:0/3432457768 >> v1:172.21.15.193:6789/0 conn(0x7fefe40a98f0 legacy=0x7fefe4133440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:42.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.484+0000 7fefebe93640 1 -- 172.21.15.66:0/3432457768 shutdown_connections 2024-03-25T18:05:42.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.484+0000 7fefebe93640 1 -- 172.21.15.66:0/3432457768 >> 172.21.15.66:0/3432457768 conn(0x7fefe40b7ea0 msgr2=0x7fefe4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:42.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.484+0000 7fefebe93640 1 -- 172.21.15.66:0/3432457768 shutdown_connections 2024-03-25T18:05:42.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.484+0000 7fefebe93640 1 -- 172.21.15.66:0/3432457768 wait complete. 2024-03-25T18:05:42.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:42.484+0000 7fefebe93640 1 librados: shutdown 2024-03-25T18:05:42.497 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:05:43.818 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:02:14.798 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:43.829 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:02:14.810 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:43.854 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:02:14.813 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:43.860 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:02:14.840 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:43.865 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:02:14.824 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:43.884 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:02:14.842 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:43.892 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:02:14.873 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:43.898 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:02:14.879 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:43.922 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:02:14.903 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:05:49.304 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:05:49.426 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.426+0000 7fc3836a8640 10 monclient: get_monmap_and_config 2024-03-25T18:05:49.426 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.426+0000 7fc3836a8640 10 monclient: build_initial_monmap 2024-03-25T18:05:49.427 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.426+0000 7fc3836a8640 10 monclient: monmap: 2024-03-25T18:05:49.427 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:05:49.427 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:49.427 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:05:49.426635+0000 2024-03-25T18:05:49.427 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:05:49.426635+0000 2024-03-25T18:05:49.427 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:49.427 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:49.427 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:05:49.427 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:05:49.427 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:05:49.427 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:49.429 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.426+0000 7fc3836a8640 1 Processor -- start 2024-03-25T18:05:49.429 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.427+0000 7fc3836a8640 1 -- start start 2024-03-25T18:05:49.429 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.427+0000 7fc3836a8640 10 monclient: init 2024-03-25T18:05:49.429 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.427+0000 7fc3836a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:49.429 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.427+0000 7fc3836a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:05:49.429 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.427+0000 7fc3836a8640 10 monclient(hunting): picked mon.noname-b con 0x7fc37c0a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:05:49.430 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.428+0000 7fc3836a8640 10 monclient(hunting): picked mon.noname-c con 0x7fc37c13f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:05:49.430 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.428+0000 7fc3836a8640 10 monclient(hunting): picked mon.noname-a con 0x7fc37c1430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:05:49.430 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.428+0000 7fc3836a8640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc37c142520 con 0x7fc37c1430c0 2024-03-25T18:05:49.430 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.428+0000 7fc3836a8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc37c0b28a0 con 0x7fc37c13f710 2024-03-25T18:05:49.430 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.428+0000 7fc3836a8640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc37c0b2a30 con 0x7fc37c0a98f0 2024-03-25T18:05:49.430 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.428+0000 7fc3836a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:49.430 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.428+0000 7fc3836a8640 10 monclient(hunting): authenticate will time out at 7098.487176s 2024-03-25T18:05:49.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.443+0000 7fc380c43640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fc37c13f710 0x7fc37c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:45984/0 (socket says 172.21.15.66:45984) 2024-03-25T18:05:49.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.443+0000 7fc380c43640 1 -- 172.21.15.66:0/978580849 learned_addr learned my addr 172.21.15.66:0/978580849 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:05:49.459 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.459+0000 7fc373fff640 1 -- 172.21.15.66:0/978580849 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 157166251 0 0) 0x7fc37c0b28a0 con 0x7fc37c13f710 2024-03-25T18:05:49.459 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.459+0000 7fc373fff640 10 monclient(hunting): _negotiate 2024-03-25T18:05:49.459 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.459+0000 7fc373fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:49.459 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.459+0000 7fc373fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:49.459 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.459+0000 7fc373fff640 10 monclient(hunting): my global_id is 4306 2024-03-25T18:05:49.459 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.459+0000 7fc373fff640 1 -- 172.21.15.66:0/978580849 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc368003520 con 0x7fc37c13f710 2024-03-25T18:05:49.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.465+0000 7fc373fff640 1 -- 172.21.15.66:0/978580849 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396930312 0 0) 0x7fc37c0b2a30 con 0x7fc37c0a98f0 2024-03-25T18:05:49.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.465+0000 7fc373fff640 10 monclient(hunting): _negotiate 2024-03-25T18:05:49.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.465+0000 7fc373fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:49.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.465+0000 7fc373fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:49.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.465+0000 7fc373fff640 10 monclient(hunting): my global_id is 4307 2024-03-25T18:05:49.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.465+0000 7fc373fff640 1 -- 172.21.15.66:0/978580849 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc37c0b28a0 con 0x7fc37c0a98f0 2024-03-25T18:05:49.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.478+0000 7fc373fff640 1 -- 172.21.15.66:0/978580849 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 293230740 0 0) 0x7fc37c142520 con 0x7fc37c1430c0 2024-03-25T18:05:49.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.478+0000 7fc373fff640 10 monclient(hunting): _negotiate 2024-03-25T18:05:49.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.478+0000 7fc373fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:49.479 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.478+0000 7fc373fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:49.479 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.478+0000 7fc373fff640 10 monclient(hunting): my global_id is 4311 2024-03-25T18:05:49.479 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.478+0000 7fc373fff640 1 -- 172.21.15.66:0/978580849 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc37c0b2a30 con 0x7fc37c1430c0 2024-03-25T18:05:49.506 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.505+0000 7fc373fff640 1 -- 172.21.15.66:0/978580849 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1058539780 0 0) 0x7fc37c0b2a30 con 0x7fc37c1430c0 2024-03-25T18:05:49.506 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.505+0000 7fc373fff640 1 -- 172.21.15.66:0/978580849 >> v1:172.21.15.66:6790/0 conn(0x7fc37c0a98f0 legacy=0x7fc37c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:49.506 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.505+0000 7fc373fff640 1 -- 172.21.15.66:0/978580849 >> v1:172.21.15.193:6789/0 conn(0x7fc37c13f710 legacy=0x7fc37c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:49.506 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.505+0000 7fc373fff640 10 monclient: _finish_hunting 0 2024-03-25T18:05:49.506 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.505+0000 7fc373fff640 1 monclient: found mon.noname-a 2024-03-25T18:05:49.506 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.505+0000 7fc373fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:49.506 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.505+0000 7fc373fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.66:6789/0 2024-03-25T18:05:49.506 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.505+0000 7fc373fff640 1 -- 172.21.15.66:0/978580849 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc37c0b44c0 con 0x7fc37c1430c0 2024-03-25T18:05:49.506 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.505+0000 7fc373fff640 10 monclient: _finish_auth 0 2024-03-25T18:05:49.506 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.505+0000 7fc373fff640 10 monclient: _check_auth_tickets 2024-03-25T18:05:49.506 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.505+0000 7fc373fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:49.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.505+0000 7fc373fff640 1 -- 172.21.15.66:0/978580849 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fc374001930 con 0x7fc37c1430c0 2024-03-25T18:05:49.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.505+0000 7fc373fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:49.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.505+0000 7fc373fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:05:49.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.505+0000 7fc373fff640 10 monclient: dump: 2024-03-25T18:05:49.507 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:49.507 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:49.507 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:49.507 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:49.507 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:49.507 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:49.507 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:49.507 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:49.507 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:49.507 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:49.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.505+0000 7fc3836a8640 5 monclient: authenticate success, global_id 4311 2024-03-25T18:05:49.508 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.505+0000 7fc3836a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:49.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.574+0000 7fc373fff640 1 -- 172.21.15.66:0/978580849 <== mon.0 v1:172.21.15.66:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc374001a90 con 0x7fc37c1430c0 2024-03-25T18:05:49.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.574+0000 7fc373fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:49.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.574+0000 7fc373fff640 1 -- 172.21.15.66:0/978580849 <== mon.0 v1:172.21.15.66:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fc374001ea0 con 0x7fc37c1430c0 2024-03-25T18:05:49.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.574+0000 7fc373fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:49.575 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.574+0000 7fc373fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:49.575 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.574+0000 7fc373fff640 10 monclient: dump: 2024-03-25T18:05:49.575 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:49.575 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:49.575 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:49.575 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:49.575 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:49.575 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:49.575 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:49.575 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:49.575 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:49.575 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:49.575 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.574+0000 7fc3836a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:49.576 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.574+0000 7fc3836a8640 4 set_mon_vals no callback set 2024-03-25T18:05:49.578 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.575+0000 7fc3836a8640 10 monclient: shutdown 2024-03-25T18:05:49.578 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.575+0000 7fc3836a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:49.578 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.575+0000 7fc3836a8640 1 -- 172.21.15.66:0/978580849 >> v1:172.21.15.66:6789/0 conn(0x7fc37c1430c0 legacy=0x7fc37c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:49.578 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.575+0000 7fc3836a8640 1 -- 172.21.15.66:0/978580849 shutdown_connections 2024-03-25T18:05:49.578 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.575+0000 7fc3836a8640 1 -- 172.21.15.66:0/978580849 >> 172.21.15.66:0/978580849 conn(0x7fc37c0b7ea0 msgr2=0x7fc37c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:49.578 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.575+0000 7fc3836a8640 1 -- 172.21.15.66:0/978580849 shutdown_connections 2024-03-25T18:05:49.578 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.575+0000 7fc3836a8640 1 -- 172.21.15.66:0/978580849 wait complete. 2024-03-25T18:05:49.578 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.576+0000 7fc3836a8640 10 monclient: build_initial_monmap 2024-03-25T18:05:49.578 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.576+0000 7fc3836a8640 10 monclient: monmap: 2024-03-25T18:05:49.579 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:05:49.579 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:49.579 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:05:49.579 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:05:49.579 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:49.579 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:05:49.579 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:05:49.579 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:05:49.579 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:05:49.579 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:49.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.576+0000 7fc3836a8640 1 librados: starting msgr at 2024-03-25T18:05:49.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.576+0000 7fc3836a8640 1 librados: starting objecter 2024-03-25T18:05:49.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.576+0000 7fc3836a8640 1 Processor -- start 2024-03-25T18:05:49.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.576+0000 7fc3836a8640 1 -- start start 2024-03-25T18:05:49.580 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.576+0000 7fc3836a8640 1 librados: setting wanted keys 2024-03-25T18:05:49.580 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.576+0000 7fc3836a8640 1 librados: calling monclient init 2024-03-25T18:05:49.580 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.576+0000 7fc3836a8640 10 monclient: init 2024-03-25T18:05:49.580 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.577+0000 7fc3836a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:49.580 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.577+0000 7fc3836a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:05:49.580 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.577+0000 7fc3836a8640 10 monclient(hunting): picked mon.noname-b con 0x7fc37c0a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:05:49.580 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.577+0000 7fc3836a8640 10 monclient(hunting): picked mon.noname-a con 0x7fc37c13f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:05:49.580 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.577+0000 7fc3836a8640 10 monclient(hunting): picked mon.noname-c con 0x7fc37c1430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:05:49.580 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.577+0000 7fc3836a8640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc37c139890 con 0x7fc37c13f710 2024-03-25T18:05:49.580 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.577+0000 7fc3836a8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc37c1dba80 con 0x7fc37c0a98f0 2024-03-25T18:05:49.580 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.577+0000 7fc3836a8640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc37c13ace0 con 0x7fc37c1430c0 2024-03-25T18:05:49.580 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.577+0000 7fc3836a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:49.580 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.577+0000 7fc3836a8640 10 monclient(hunting): authenticate will time out at 7098.636722s 2024-03-25T18:05:49.584 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.584+0000 7fc381444640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fc37c0a98f0 0x7fc37c139d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:46074/0 (socket says 172.21.15.66:46074) 2024-03-25T18:05:49.584 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.584+0000 7fc381444640 1 -- 172.21.15.66:0/2413899716 learned_addr learned my addr 172.21.15.66:0/2413899716 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:05:49.588 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.588+0000 7fc371ffb640 10 client.?.objecter ms_handle_connect 0x7fc37c0a98f0 2024-03-25T18:05:49.589 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.588+0000 7fc371ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:49.597 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.597+0000 7fc371ffb640 10 client.?.objecter ms_handle_connect 0x7fc37c13f710 2024-03-25T18:05:49.597 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.597+0000 7fc371ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:49.598 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.597+0000 7fc371ffb640 1 -- 172.21.15.66:0/2413899716 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3415164597 0 0) 0x7fc37c1dba80 con 0x7fc37c0a98f0 2024-03-25T18:05:49.598 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.597+0000 7fc371ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:05:49.598 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.597+0000 7fc371ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:49.598 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.597+0000 7fc371ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:49.598 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.597+0000 7fc371ffb640 10 monclient(hunting): my global_id is 4309 2024-03-25T18:05:49.598 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.597+0000 7fc371ffb640 1 -- 172.21.15.66:0/2413899716 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc360004eb0 con 0x7fc37c0a98f0 2024-03-25T18:05:49.598 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.597+0000 7fc371ffb640 10 client.?.objecter ms_handle_connect 0x7fc37c1430c0 2024-03-25T18:05:49.598 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.597+0000 7fc371ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:49.610 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.609+0000 7fc371ffb640 1 -- 172.21.15.66:0/2413899716 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885660018 0 0) 0x7fc37c13ace0 con 0x7fc37c1430c0 2024-03-25T18:05:49.610 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.609+0000 7fc371ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:05:49.610 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.609+0000 7fc371ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:49.610 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.609+0000 7fc371ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:49.610 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.609+0000 7fc371ffb640 10 monclient(hunting): my global_id is 4310 2024-03-25T18:05:49.610 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.609+0000 7fc371ffb640 1 -- 172.21.15.66:0/2413899716 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc37c1dba80 con 0x7fc37c1430c0 2024-03-25T18:05:49.612 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.611+0000 7fc371ffb640 1 -- 172.21.15.66:0/2413899716 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4035036137 0 0) 0x7fc360004eb0 con 0x7fc37c0a98f0 2024-03-25T18:05:49.612 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.611+0000 7fc371ffb640 1 -- 172.21.15.66:0/2413899716 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc37c13ace0 con 0x7fc37c0a98f0 2024-03-25T18:05:49.612 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.611+0000 7fc371ffb640 1 -- 172.21.15.66:0/2413899716 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fc378001a10 con 0x7fc37c0a98f0 2024-03-25T18:05:49.612 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.611+0000 7fc371ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:49.612 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.611+0000 7fc371ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:05:49.612 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.611+0000 7fc371ffb640 10 monclient(hunting): dump: 2024-03-25T18:05:49.613 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:49.613 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:49.613 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:49.613 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:49.613 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:49.613 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:49.613 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:49.613 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:49.613 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:49.613 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:49.613 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.612+0000 7fc371ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:49.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.618+0000 7fc371ffb640 1 -- 172.21.15.66:0/2413899716 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1702390946 0 0) 0x7fc37c139890 con 0x7fc37c13f710 2024-03-25T18:05:49.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.618+0000 7fc371ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:05:49.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.618+0000 7fc371ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:49.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.618+0000 7fc371ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:49.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.618+0000 7fc371ffb640 10 monclient(hunting): my global_id is 4314 2024-03-25T18:05:49.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.618+0000 7fc371ffb640 1 -- 172.21.15.66:0/2413899716 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc360004eb0 con 0x7fc37c13f710 2024-03-25T18:05:49.621 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.620+0000 7fc371ffb640 1 -- 172.21.15.66:0/2413899716 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2699661011 0 0) 0x7fc37c13ace0 con 0x7fc37c0a98f0 2024-03-25T18:05:49.621 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.620+0000 7fc371ffb640 1 -- 172.21.15.66:0/2413899716 >> v1:172.21.15.66:6790/0 conn(0x7fc37c1430c0 legacy=0x7fc37c1db370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:49.622 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.620+0000 7fc371ffb640 1 -- 172.21.15.66:0/2413899716 >> v1:172.21.15.66:6789/0 conn(0x7fc37c13f710 legacy=0x7fc37c1dac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:49.622 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.620+0000 7fc371ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:05:49.622 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.620+0000 7fc371ffb640 1 monclient: found mon.b 2024-03-25T18:05:49.622 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.620+0000 7fc371ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:49.622 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.621+0000 7fc371ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:05:49.622 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.621+0000 7fc371ffb640 1 -- 172.21.15.66:0/2413899716 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc37c13afc0 con 0x7fc37c0a98f0 2024-03-25T18:05:49.622 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.621+0000 7fc371ffb640 10 monclient: _finish_auth 0 2024-03-25T18:05:49.622 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.621+0000 7fc371ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:05:49.622 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.621+0000 7fc371ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:49.622 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.621+0000 7fc3836a8640 5 monclient: authenticate success, global_id 4309 2024-03-25T18:05:49.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.621+0000 7fc3836a8640 10 monclient: _renew_subs 2024-03-25T18:05:49.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.621+0000 7fc3836a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:05:49.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.621+0000 7fc3836a8640 1 -- 172.21.15.66:0/2413899716 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc37c13b2a0 con 0x7fc37c0a98f0 2024-03-25T18:05:49.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.621+0000 7fc3836a8640 10 client.4309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:49.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.621+0000 7fc3836a8640 10 monclient: _renew_subs 2024-03-25T18:05:49.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.621+0000 7fc3836a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:05:49.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.621+0000 7fc3836a8640 1 -- 172.21.15.66:0/2413899716 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc37c0b7b50 con 0x7fc37c0a98f0 2024-03-25T18:05:49.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.621+0000 7fc3836a8640 1 librados: init done 2024-03-25T18:05:49.624 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.622+0000 7fc3836a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:49.624 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.622+0000 7fc3836a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:49.624 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.622+0000 7fc3836a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:05:49.624 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.622+0000 7fc3836a8640 1 -- 172.21.15.66:0/2413899716 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc37c0627b0 con 0x7fc37c0a98f0 2024-03-25T18:05:49.901 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.901+0000 7fc371ffb640 1 -- 172.21.15.66:0/2413899716 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc378001c50 con 0x7fc37c0a98f0 2024-03-25T18:05:49.902 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.901+0000 7fc371ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:49.902 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.901+0000 7fc371ffb640 1 -- 172.21.15.66:0/2413899716 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fc378003a60 con 0x7fc37c0a98f0 2024-03-25T18:05:49.902 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.901+0000 7fc371ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:49.902 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.901+0000 7fc371ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:49.902 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.901+0000 7fc371ffb640 10 monclient: dump: 2024-03-25T18:05:49.902 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:49.902 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:49.902 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:49.902 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:49.902 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:49.903 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:49.903 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:49.903 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:49.903 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:49.903 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:49.903 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.901+0000 7fc3737fe640 4 set_mon_vals no callback set 2024-03-25T18:05:49.903 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.902+0000 7fc371ffb640 1 -- 172.21.15.66:0/2413899716 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7fc378003c80 con 0x7fc37c0a98f0 2024-03-25T18:05:49.904 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.902+0000 7fc380c43640 10 monclient: get_auth_request con 0x7fc36006de00 auth_method 0 2024-03-25T18:05:49.904 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.903+0000 7fc371ffb640 10 client.4309.objecter ms_handle_connect 0x7fc36006de00 2024-03-25T18:05:49.907 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.907+0000 7fc371ffb640 1 -- 172.21.15.66:0/2413899716 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7fc3780547e0 con 0x7fc37c0a98f0 2024-03-25T18:05:49.908 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.907+0000 7fc371ffb640 10 client.4309.objecter ms_dispatch 0x7fc37c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:05:49.908 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.907+0000 7fc371ffb640 3 client.4309.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:05:49.908 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.907+0000 7fc371ffb640 3 client.4309.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:05:49.908 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.907+0000 7fc371ffb640 20 client.4309.objecter dump_active .. 0 homeless 2024-03-25T18:05:49.908 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.908+0000 7fc371ffb640 1 -- 172.21.15.66:0/2413899716 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc3780502f0 con 0x7fc37c0a98f0 2024-03-25T18:05:49.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.908+0000 7fc371ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:49.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:49.908+0000 7fc371ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:05:50.167 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:50.166+0000 7fc3836a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:50.167 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:50.166+0000 7fc3836a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:50.167 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:50.166+0000 7fc3836a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:05:50.167 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:50.166+0000 7fc3836a8640 1 -- 172.21.15.66:0/2413899716 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc37c14f320 con 0x7fc37c0a98f0 2024-03-25T18:05:50.193 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:50.193+0000 7fc371ffb640 1 -- 172.21.15.66:0/2413899716 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7fc378052220 con 0x7fc37c0a98f0 2024-03-25T18:05:50.193 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:50.193+0000 7fc371ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:50.193 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:50.193+0000 7fc371ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:05:50.194 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:05:50.194 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:05:50.196 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:50.195+0000 7fc3836a8640 10 librados: watch_flush enter 2024-03-25T18:05:50.196 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:50.195+0000 7fc3836a8640 10 librados: watch_flush exit 2024-03-25T18:05:50.196 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:50.195+0000 7fc3836a8640 20 client.4309.objecter shutdown clearing up homeless session... 2024-03-25T18:05:50.197 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:50.195+0000 7fc3836a8640 10 client.4309.objecter successfully canceled tick 2024-03-25T18:05:50.197 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:50.195+0000 7fc3836a8640 1 -- 172.21.15.66:0/2413899716 >> v1:172.21.15.193:6800/2275676803 conn(0x7fc36006de00 legacy=0x7fc3600702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:50.197 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:50.195+0000 7fc3836a8640 10 monclient: shutdown 2024-03-25T18:05:50.197 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:50.195+0000 7fc3836a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:50.197 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:50.195+0000 7fc3836a8640 1 -- 172.21.15.66:0/2413899716 >> v1:172.21.15.193:6789/0 conn(0x7fc37c0a98f0 legacy=0x7fc37c139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:50.197 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:50.195+0000 7fc3836a8640 1 -- 172.21.15.66:0/2413899716 shutdown_connections 2024-03-25T18:05:50.197 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:50.195+0000 7fc3836a8640 1 -- 172.21.15.66:0/2413899716 >> 172.21.15.66:0/2413899716 conn(0x7fc37c0b7ea0 msgr2=0x7fc37c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:50.197 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:50.195+0000 7fc3836a8640 1 -- 172.21.15.66:0/2413899716 shutdown_connections 2024-03-25T18:05:50.197 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:50.196+0000 7fc3836a8640 1 -- 172.21.15.66:0/2413899716 wait complete. 2024-03-25T18:05:50.197 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:50.196+0000 7fc3836a8640 1 librados: shutdown 2024-03-25T18:05:50.207 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:05:57.015 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:05:57.135 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.134+0000 7fd80308f640 10 monclient: get_monmap_and_config 2024-03-25T18:05:57.135 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.134+0000 7fd80308f640 10 monclient: build_initial_monmap 2024-03-25T18:05:57.135 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.134+0000 7fd80308f640 10 monclient: monmap: 2024-03-25T18:05:57.135 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:05:57.135 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:57.135 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:05:57.135248+0000 2024-03-25T18:05:57.135 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:05:57.135248+0000 2024-03-25T18:05:57.135 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:57.136 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:57.136 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:05:57.136 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:05:57.136 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:05:57.136 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:57.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.135+0000 7fd80308f640 1 Processor -- start 2024-03-25T18:05:57.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.135+0000 7fd80308f640 1 -- start start 2024-03-25T18:05:57.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.135+0000 7fd80308f640 10 monclient: init 2024-03-25T18:05:57.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.136+0000 7fd80308f640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:57.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.136+0000 7fd80308f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:05:57.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.136+0000 7fd80308f640 10 monclient(hunting): picked mon.noname-b con 0x7fd7fc0a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:05:57.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.136+0000 7fd80308f640 10 monclient(hunting): picked mon.noname-a con 0x7fd7fc13f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:05:57.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.136+0000 7fd80308f640 10 monclient(hunting): picked mon.noname-c con 0x7fd7fc1430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:05:57.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.136+0000 7fd80308f640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7fc142520 con 0x7fd7fc13f710 2024-03-25T18:05:57.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.136+0000 7fd80308f640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7fc0b28a0 con 0x7fd7fc1430c0 2024-03-25T18:05:57.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.136+0000 7fd80308f640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7fc0b2a30 con 0x7fd7fc0a98f0 2024-03-25T18:05:57.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.136+0000 7fd80308f640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:57.139 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.136+0000 7fd80308f640 10 monclient(hunting): authenticate will time out at 7106.195748s 2024-03-25T18:05:57.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.143+0000 7fd800e2b640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7fd7fc0a98f0 0x7fd7fc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:46708/0 (socket says 172.21.15.66:46708) 2024-03-25T18:05:57.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.144+0000 7fd800e2b640 1 -- 172.21.15.66:0/2689495734 learned_addr learned my addr 172.21.15.66:0/2689495734 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:05:57.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.154+0000 7fd7fb7fe640 1 -- 172.21.15.66:0/2689495734 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307203320 0 0) 0x7fd7fc0b28a0 con 0x7fd7fc1430c0 2024-03-25T18:05:57.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.154+0000 7fd7fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:05:57.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.154+0000 7fd7fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:57.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.154+0000 7fd7fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:57.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.154+0000 7fd7fb7fe640 10 monclient(hunting): my global_id is 4312 2024-03-25T18:05:57.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.154+0000 7fd7fb7fe640 1 -- 172.21.15.66:0/2689495734 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7ec003520 con 0x7fd7fc1430c0 2024-03-25T18:05:57.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.160+0000 7fd7fb7fe640 1 -- 172.21.15.66:0/2689495734 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1769251604 0 0) 0x7fd7fc0b2a30 con 0x7fd7fc0a98f0 2024-03-25T18:05:57.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.160+0000 7fd7fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:05:57.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.160+0000 7fd7fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:57.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.160+0000 7fd7fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:57.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.160+0000 7fd7fb7fe640 10 monclient(hunting): my global_id is 4313 2024-03-25T18:05:57.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.160+0000 7fd7fb7fe640 1 -- 172.21.15.66:0/2689495734 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7fc0b28a0 con 0x7fd7fc0a98f0 2024-03-25T18:05:57.169 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.168+0000 7fd7fb7fe640 1 -- 172.21.15.66:0/2689495734 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3347774726 0 0) 0x7fd7ec003520 con 0x7fd7fc1430c0 2024-03-25T18:05:57.169 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.168+0000 7fd7fb7fe640 1 -- 172.21.15.66:0/2689495734 >> v1:172.21.15.66:6790/0 conn(0x7fd7fc0a98f0 legacy=0x7fd7fc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:57.169 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.168+0000 7fd7fb7fe640 1 -- 172.21.15.66:0/2689495734 >> v1:172.21.15.66:6789/0 conn(0x7fd7fc13f710 legacy=0x7fd7fc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:57.169 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.168+0000 7fd7fb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:05:57.169 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.168+0000 7fd7fb7fe640 1 monclient: found mon.noname-c 2024-03-25T18:05:57.169 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.168+0000 7fd7fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:57.169 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.168+0000 7fd7fb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:05:57.169 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.168+0000 7fd7fb7fe640 1 -- 172.21.15.66:0/2689495734 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7fc0b44c0 con 0x7fd7fc1430c0 2024-03-25T18:05:57.170 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.168+0000 7fd7fb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:05:57.170 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.168+0000 7fd7fb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:05:57.170 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.168+0000 7fd7fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:57.170 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.168+0000 7fd7fb7fe640 1 -- 172.21.15.66:0/2689495734 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fd7f4001a10 con 0x7fd7fc1430c0 2024-03-25T18:05:57.170 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.168+0000 7fd80308f640 5 monclient: authenticate success, global_id 4312 2024-03-25T18:05:57.170 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.168+0000 7fd80308f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:57.170 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.168+0000 7fd7fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:57.170 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.168+0000 7fd7fb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:05:57.170 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.168+0000 7fd7fb7fe640 10 monclient: dump: 2024-03-25T18:05:57.170 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:57.170 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:57.171 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:57.171 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:57.171 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:57.171 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:57.171 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:57.171 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:57.171 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:57.171 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:57.171 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.168+0000 7fd80308f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:05:57.176 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.175+0000 7fd7fb7fe640 1 -- 172.21.15.66:0/2689495734 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7f4001b70 con 0x7fd7fc1430c0 2024-03-25T18:05:57.176 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.175+0000 7fd7fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:57.176 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.175+0000 7fd7fb7fe640 1 -- 172.21.15.66:0/2689495734 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fd7f4001f80 con 0x7fd7fc1430c0 2024-03-25T18:05:57.176 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.175+0000 7fd80308f640 10 monclient: get_monmap_and_config success 2024-03-25T18:05:57.176 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.175+0000 7fd80308f640 4 set_mon_vals no callback set 2024-03-25T18:05:57.176 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.175+0000 7fd7fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:57.176 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.175+0000 7fd7fb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:57.176 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.175+0000 7fd7fb7fe640 10 monclient: dump: 2024-03-25T18:05:57.176 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:57.176 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:57.176 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:57.176 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:57.177 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:57.177 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:57.177 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:57.177 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:57.177 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:57.177 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:57.180 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.176+0000 7fd80308f640 10 monclient: shutdown 2024-03-25T18:05:57.180 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.176+0000 7fd80308f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:57.180 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.176+0000 7fd80308f640 1 -- 172.21.15.66:0/2689495734 >> v1:172.21.15.193:6789/0 conn(0x7fd7fc1430c0 legacy=0x7fd7fc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:57.180 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.176+0000 7fd80308f640 1 -- 172.21.15.66:0/2689495734 shutdown_connections 2024-03-25T18:05:57.180 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.176+0000 7fd80308f640 1 -- 172.21.15.66:0/2689495734 >> 172.21.15.66:0/2689495734 conn(0x7fd7fc0b7ea0 msgr2=0x7fd7fc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:57.180 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.176+0000 7fd80308f640 1 -- 172.21.15.66:0/2689495734 shutdown_connections 2024-03-25T18:05:57.180 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.176+0000 7fd80308f640 1 -- 172.21.15.66:0/2689495734 wait complete. 2024-03-25T18:05:57.180 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.177+0000 7fd80308f640 10 monclient: build_initial_monmap 2024-03-25T18:05:57.180 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.177+0000 7fd80308f640 10 monclient: monmap: 2024-03-25T18:05:57.180 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:05:57.181 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:05:57.181 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:05:57.181 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:05:57.181 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:05:57.181 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:05:57.181 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:05:57.181 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:05:57.181 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:05:57.181 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:57.181 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.178+0000 7fd80308f640 1 librados: starting msgr at 2024-03-25T18:05:57.181 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.178+0000 7fd80308f640 1 librados: starting objecter 2024-03-25T18:05:57.181 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.178+0000 7fd80308f640 1 Processor -- start 2024-03-25T18:05:57.181 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.178+0000 7fd80308f640 1 -- start start 2024-03-25T18:05:57.181 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.178+0000 7fd80308f640 1 librados: setting wanted keys 2024-03-25T18:05:57.182 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.178+0000 7fd80308f640 1 librados: calling monclient init 2024-03-25T18:05:57.182 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.178+0000 7fd80308f640 10 monclient: init 2024-03-25T18:05:57.182 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.179+0000 7fd80308f640 10 monclient: _reopen_session rank -1 2024-03-25T18:05:57.182 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.179+0000 7fd80308f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:05:57.182 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.179+0000 7fd80308f640 10 monclient(hunting): picked mon.noname-a con 0x7fd7fc0a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:05:57.182 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.179+0000 7fd80308f640 10 monclient(hunting): picked mon.noname-c con 0x7fd7fc13f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:05:57.182 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.179+0000 7fd80308f640 10 monclient(hunting): picked mon.noname-b con 0x7fd7fc1430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:05:57.182 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.179+0000 7fd80308f640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7fc137c60 con 0x7fd7fc0a98f0 2024-03-25T18:05:57.182 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.179+0000 7fd80308f640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7fc139160 con 0x7fd7fc1430c0 2024-03-25T18:05:57.182 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.179+0000 7fd80308f640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7fc1380f0 con 0x7fd7fc13f710 2024-03-25T18:05:57.182 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.179+0000 7fd80308f640 10 monclient(hunting): _renew_subs 2024-03-25T18:05:57.182 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.179+0000 7fd80308f640 10 monclient(hunting): authenticate will time out at 7106.238609s 2024-03-25T18:05:57.185 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.184+0000 7fd800e2b640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7fd7fc0a98f0 0x7fd7fc1dab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:59442/0 (socket says 172.21.15.66:59442) 2024-03-25T18:05:57.185 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.184+0000 7fd800e2b640 1 -- 172.21.15.66:0/3358607557 learned_addr learned my addr 172.21.15.66:0/3358607557 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:05:57.189 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.189+0000 7fd7f97fa640 10 client.?.objecter ms_handle_connect 0x7fd7fc1430c0 2024-03-25T18:05:57.189 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.189+0000 7fd7f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:57.193 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.193+0000 7fd7f97fa640 10 client.?.objecter ms_handle_connect 0x7fd7fc0a98f0 2024-03-25T18:05:57.193 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.193+0000 7fd7f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:57.200 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.199+0000 7fd7f97fa640 10 client.?.objecter ms_handle_connect 0x7fd7fc13f710 2024-03-25T18:05:57.200 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.199+0000 7fd7f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:05:57.208 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.208+0000 7fd7f97fa640 1 -- 172.21.15.66:0/3358607557 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602959289 0 0) 0x7fd7fc137c60 con 0x7fd7fc0a98f0 2024-03-25T18:05:57.208 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.208+0000 7fd7f97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:05:57.208 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.208+0000 7fd7f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:57.208 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.208+0000 7fd7f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:57.209 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.208+0000 7fd7f97fa640 10 monclient(hunting): my global_id is 4320 2024-03-25T18:05:57.209 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.208+0000 7fd7f97fa640 1 -- 172.21.15.66:0/3358607557 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7e0004eb0 con 0x7fd7fc0a98f0 2024-03-25T18:05:57.210 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.209+0000 7fd7f97fa640 1 -- 172.21.15.66:0/3358607557 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 331398633 0 0) 0x7fd7fc139160 con 0x7fd7fc1430c0 2024-03-25T18:05:57.210 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.209+0000 7fd7f97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:05:57.210 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.209+0000 7fd7f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:57.210 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.209+0000 7fd7f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:57.210 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.209+0000 7fd7f97fa640 10 monclient(hunting): my global_id is 4315 2024-03-25T18:05:57.210 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.209+0000 7fd7f97fa640 1 -- 172.21.15.66:0/3358607557 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7fc137c60 con 0x7fd7fc1430c0 2024-03-25T18:05:57.218 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.217+0000 7fd7f97fa640 1 -- 172.21.15.66:0/3358607557 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 594701838 0 0) 0x7fd7fc137c60 con 0x7fd7fc1430c0 2024-03-25T18:05:57.218 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.217+0000 7fd7f97fa640 1 -- 172.21.15.66:0/3358607557 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7fc139160 con 0x7fd7fc1430c0 2024-03-25T18:05:57.218 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.217+0000 7fd7f97fa640 1 -- 172.21.15.66:0/3358607557 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fd7f4001e80 con 0x7fd7fc1430c0 2024-03-25T18:05:57.218 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.217+0000 7fd7f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:57.218 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.217+0000 7fd7f97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:05:57.218 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.217+0000 7fd7f97fa640 10 monclient(hunting): dump: 2024-03-25T18:05:57.218 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:57.218 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:57.218 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:57.218 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:57.219 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:57.219 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:57.219 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:57.219 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:57.219 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:57.219 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:57.219 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.217+0000 7fd7f97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:05:57.219 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.219+0000 7fd7f97fa640 1 -- 172.21.15.66:0/3358607557 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1256625585 0 0) 0x7fd7fc1380f0 con 0x7fd7fc13f710 2024-03-25T18:05:57.220 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.219+0000 7fd7f97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:05:57.220 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.219+0000 7fd7f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:05:57.220 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.219+0000 7fd7f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:05:57.220 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.219+0000 7fd7f97fa640 10 monclient(hunting): my global_id is 4316 2024-03-25T18:05:57.220 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.219+0000 7fd7f97fa640 1 -- 172.21.15.66:0/3358607557 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7fc137c60 con 0x7fd7fc13f710 2024-03-25T18:05:57.232 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.232+0000 7fd7f97fa640 1 -- 172.21.15.66:0/3358607557 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 554943865 0 0) 0x7fd7e0004eb0 con 0x7fd7fc0a98f0 2024-03-25T18:05:57.232 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.232+0000 7fd7f97fa640 1 -- 172.21.15.66:0/3358607557 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7fc1380f0 con 0x7fd7fc0a98f0 2024-03-25T18:05:57.232 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.232+0000 7fd7f97fa640 1 -- 172.21.15.66:0/3358607557 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fd7f00019b0 con 0x7fd7fc0a98f0 2024-03-25T18:05:57.232 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.232+0000 7fd7f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:57.232 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.232+0000 7fd7f97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:05:57.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.232+0000 7fd7f97fa640 10 monclient(hunting): dump: 2024-03-25T18:05:57.233 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:57.233 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:57.233 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:57.233 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:57.233 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:57.233 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:57.233 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:57.233 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:57.233 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:57.233 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:57.239 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.238+0000 7fd7f97fa640 1 -- 172.21.15.66:0/3358607557 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4116629728 0 0) 0x7fd7fc139160 con 0x7fd7fc1430c0 2024-03-25T18:05:57.239 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.239+0000 7fd7f97fa640 1 -- 172.21.15.66:0/3358607557 >> v1:172.21.15.66:6790/0 conn(0x7fd7fc13f710 legacy=0x7fd7fc133070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:57.239 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.239+0000 7fd7f97fa640 1 -- 172.21.15.66:0/3358607557 >> v1:172.21.15.66:6789/0 conn(0x7fd7fc0a98f0 legacy=0x7fd7fc1dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:57.240 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.239+0000 7fd7f97fa640 10 monclient: _finish_hunting 0 2024-03-25T18:05:57.240 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.239+0000 7fd7f97fa640 1 monclient: found mon.b 2024-03-25T18:05:57.240 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.239+0000 7fd7f97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:05:57.240 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.239+0000 7fd7f97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:05:57.240 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.239+0000 7fd7f97fa640 1 -- 172.21.15.66:0/3358607557 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7fc1397f0 con 0x7fd7fc1430c0 2024-03-25T18:05:57.240 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.239+0000 7fd7f97fa640 10 monclient: _finish_auth 0 2024-03-25T18:05:57.240 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.239+0000 7fd7f97fa640 10 monclient: _check_auth_tickets 2024-03-25T18:05:57.240 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.239+0000 7fd7f97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:05:57.240 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.239+0000 7fd80308f640 5 monclient: authenticate success, global_id 4315 2024-03-25T18:05:57.240 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.239+0000 7fd80308f640 10 monclient: _renew_subs 2024-03-25T18:05:57.240 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.239+0000 7fd80308f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:05:57.241 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.239+0000 7fd80308f640 1 -- 172.21.15.66:0/3358607557 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7fc1db280 con 0x7fd7fc1430c0 2024-03-25T18:05:57.241 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.239+0000 7fd80308f640 10 client.4315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:05:57.241 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.239+0000 7fd80308f640 10 monclient: _renew_subs 2024-03-25T18:05:57.241 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.239+0000 7fd80308f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:05:57.241 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.239+0000 7fd80308f640 1 -- 172.21.15.66:0/3358607557 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7fc133960 con 0x7fd7fc1430c0 2024-03-25T18:05:57.241 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.239+0000 7fd80308f640 1 librados: init done 2024-03-25T18:05:57.242 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.240+0000 7fd80308f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:57.242 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.240+0000 7fd80308f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:57.242 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.240+0000 7fd80308f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:05:57.242 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.240+0000 7fd80308f640 1 -- 172.21.15.66:0/3358607557 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7fc1db5d0 con 0x7fd7fc1430c0 2024-03-25T18:05:57.524 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.523+0000 7fd7f97fa640 1 -- 172.21.15.66:0/3358607557 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7f4002040 con 0x7fd7fc1430c0 2024-03-25T18:05:57.524 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.523+0000 7fd7f97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:05:57.524 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.523+0000 7fd7f97fa640 1 -- 172.21.15.66:0/3358607557 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fd7f4003dc0 con 0x7fd7fc1430c0 2024-03-25T18:05:57.524 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.523+0000 7fd7f97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:05:57.524 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.523+0000 7fd7f97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:05:57.524 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.523+0000 7fd7f97fa640 10 monclient: dump: 2024-03-25T18:05:57.524 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:05:57.524 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:05:57.524 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:57.524 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:05:57.524 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:05:57.524 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:05:57.524 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:05:57.525 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:05:57.525 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:05:57.525 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:05:57.525 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.523+0000 7fd7faffd640 4 set_mon_vals no callback set 2024-03-25T18:05:57.525 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.524+0000 7fd7f97fa640 1 -- 172.21.15.66:0/3358607557 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7fd7f4019d90 con 0x7fd7fc1430c0 2024-03-25T18:05:57.525 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.525+0000 7fd800e2b640 10 monclient: get_auth_request con 0x7fd7e006dfb0 auth_method 0 2024-03-25T18:05:57.525 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.525+0000 7fd7f97fa640 10 client.4315.objecter ms_handle_connect 0x7fd7e006dfb0 2024-03-25T18:05:57.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.530+0000 7fd7f97fa640 1 -- 172.21.15.66:0/3358607557 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7fd7f4053930 con 0x7fd7fc1430c0 2024-03-25T18:05:57.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.530+0000 7fd7f97fa640 10 client.4315.objecter ms_dispatch 0x7fd7fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:05:57.531 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.530+0000 7fd7f97fa640 3 client.4315.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:05:57.531 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.530+0000 7fd7f97fa640 3 client.4315.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:05:57.531 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.530+0000 7fd7f97fa640 20 client.4315.objecter dump_active .. 0 homeless 2024-03-25T18:05:57.531 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.531+0000 7fd7f97fa640 1 -- 172.21.15.66:0/3358607557 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd7f404f440 con 0x7fd7fc1430c0 2024-03-25T18:05:57.531 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.531+0000 7fd7f97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:05:57.531 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.531+0000 7fd7f97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:05:57.756 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.756+0000 7fd80308f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:57.756 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.756+0000 7fd80308f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:57.756 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.756+0000 7fd80308f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:05:57.757 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.756+0000 7fd80308f640 1 -- 172.21.15.66:0/3358607557 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7fc137e10 con 0x7fd7fc1430c0 2024-03-25T18:05:57.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.784+0000 7fd7f97fa640 1 -- 172.21.15.66:0/3358607557 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7fd7fc137e10 con 0x7fd7fc1430c0 2024-03-25T18:05:57.785 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.784+0000 7fd7f97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:05:57.785 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.784+0000 7fd7f97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:05:57.785 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:05:57.785 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:05:57.788 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.787+0000 7fd80308f640 10 librados: watch_flush enter 2024-03-25T18:05:57.788 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.787+0000 7fd80308f640 10 librados: watch_flush exit 2024-03-25T18:05:57.788 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.787+0000 7fd80308f640 20 client.4315.objecter shutdown clearing up homeless session... 2024-03-25T18:05:57.788 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.787+0000 7fd80308f640 10 client.4315.objecter successfully canceled tick 2024-03-25T18:05:57.788 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.787+0000 7fd80308f640 1 -- 172.21.15.66:0/3358607557 >> v1:172.21.15.193:6800/2275676803 conn(0x7fd7e006dfb0 legacy=0x7fd7e0070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:57.788 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.787+0000 7fd80308f640 10 monclient: shutdown 2024-03-25T18:05:57.788 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.787+0000 7fd80308f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:05:57.788 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.787+0000 7fd80308f640 1 -- 172.21.15.66:0/3358607557 >> v1:172.21.15.193:6789/0 conn(0x7fd7fc1430c0 legacy=0x7fd7fc1e2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:05:57.788 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.787+0000 7fd80308f640 1 -- 172.21.15.66:0/3358607557 shutdown_connections 2024-03-25T18:05:57.788 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.787+0000 7fd80308f640 1 -- 172.21.15.66:0/3358607557 >> 172.21.15.66:0/3358607557 conn(0x7fd7fc0b7ea0 msgr2=0x7fd7fc13d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:05:57.789 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.787+0000 7fd80308f640 1 -- 172.21.15.66:0/3358607557 shutdown_connections 2024-03-25T18:05:57.789 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.787+0000 7fd80308f640 1 -- 172.21.15.66:0/3358607557 wait complete. 2024-03-25T18:05:57.789 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:05:57.787+0000 7fd80308f640 1 librados: shutdown 2024-03-25T18:05:57.799 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:06:03.843 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:02:34.824 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:03.900 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:02:34.881 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:03.937 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:02:34.917 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:03.951 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:02:34.910 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:03.953 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:02:34.934 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:03.958 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:02:34.917 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:03.966 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:02:34.947 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:03.983 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:02:34.941 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:04.007 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:02:34.987 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:04.605 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:06:04.722 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.721+0000 7f402924a640 10 monclient: get_monmap_and_config 2024-03-25T18:06:04.722 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.721+0000 7f402924a640 10 monclient: build_initial_monmap 2024-03-25T18:06:04.722 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.721+0000 7f402924a640 10 monclient: monmap: 2024-03-25T18:06:04.722 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:06:04.722 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:04.723 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:06:04.722277+0000 2024-03-25T18:06:04.723 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:06:04.722277+0000 2024-03-25T18:06:04.723 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:04.723 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:04.723 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:06:04.723 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:06:04.723 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:06:04.723 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:04.725 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.722+0000 7f402924a640 1 Processor -- start 2024-03-25T18:06:04.725 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.722+0000 7f402924a640 1 -- start start 2024-03-25T18:06:04.725 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.722+0000 7f402924a640 10 monclient: init 2024-03-25T18:06:04.725 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.723+0000 7f402924a640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:04.725 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.723+0000 7f402924a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:06:04.725 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.723+0000 7f402924a640 10 monclient(hunting): picked mon.noname-a con 0x7f40240a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:06:04.725 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.723+0000 7f402924a640 10 monclient(hunting): picked mon.noname-b con 0x7f402413f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:06:04.725 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.723+0000 7f402924a640 10 monclient(hunting): picked mon.noname-c con 0x7f40241430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:06:04.725 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.723+0000 7f402924a640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4024142520 con 0x7f40240a98f0 2024-03-25T18:06:04.726 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.723+0000 7f402924a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40240b28a0 con 0x7f40241430c0 2024-03-25T18:06:04.726 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.723+0000 7f402924a640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40240b2a30 con 0x7f402413f710 2024-03-25T18:06:04.726 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.723+0000 7f402924a640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:04.726 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.723+0000 7f402924a640 10 monclient(hunting): authenticate will time out at 7113.782465s 2024-03-25T18:06:04.727 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.727+0000 7f401bfff640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f402413f710 0x7f40241429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:45328/0 (socket says 172.21.15.66:45328) 2024-03-25T18:06:04.727 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.727+0000 7f401bfff640 1 -- 172.21.15.66:0/4016251368 learned_addr learned my addr 172.21.15.66:0/4016251368 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:06:04.736 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.736+0000 7f402259c640 1 -- 172.21.15.66:0/4016251368 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177638946 0 0) 0x7f40240b2a30 con 0x7f402413f710 2024-03-25T18:06:04.736 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.736+0000 7f402259c640 10 monclient(hunting): _negotiate 2024-03-25T18:06:04.736 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.736+0000 7f402259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:04.736 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.736+0000 7f402259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:04.736 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.736+0000 7f402259c640 10 monclient(hunting): my global_id is 4319 2024-03-25T18:06:04.737 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.736+0000 7f402259c640 1 -- 172.21.15.66:0/4016251368 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4010003520 con 0x7f402413f710 2024-03-25T18:06:04.748 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.747+0000 7f402259c640 1 -- 172.21.15.66:0/4016251368 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4209697398 0 0) 0x7f4010003520 con 0x7f402413f710 2024-03-25T18:06:04.748 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.747+0000 7f402259c640 1 -- 172.21.15.66:0/4016251368 >> v1:172.21.15.193:6789/0 conn(0x7f40241430c0 legacy=0x7f402414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:04.748 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.747+0000 7f402259c640 1 -- 172.21.15.66:0/4016251368 >> v1:172.21.15.66:6789/0 conn(0x7f40240a98f0 legacy=0x7f40240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:06:04.748 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.747+0000 7f402259c640 10 monclient: _finish_hunting 0 2024-03-25T18:06:04.748 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.747+0000 7f402259c640 1 monclient: found mon.noname-b 2024-03-25T18:06:04.748 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.747+0000 7f402259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:04.748 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.747+0000 7f402259c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.66:6790/0 2024-03-25T18:06:04.749 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.747+0000 7f402259c640 1 -- 172.21.15.66:0/4016251368 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40240b44c0 con 0x7f402413f710 2024-03-25T18:06:04.749 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.747+0000 7f402259c640 10 monclient: _finish_auth 0 2024-03-25T18:06:04.749 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.747+0000 7f402259c640 10 monclient: _check_auth_tickets 2024-03-25T18:06:04.749 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.747+0000 7f402259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:04.749 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.747+0000 7f402924a640 5 monclient: authenticate success, global_id 4319 2024-03-25T18:06:04.749 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.747+0000 7f402924a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:04.750 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.748+0000 7f402259c640 1 -- 172.21.15.66:0/4016251368 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f400c001690 con 0x7f402413f710 2024-03-25T18:06:04.750 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.748+0000 7f402259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:04.750 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.748+0000 7f402259c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:06:04.750 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.748+0000 7f402259c640 10 monclient: dump: 2024-03-25T18:06:04.750 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:04.751 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:04.751 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:04.751 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:04.751 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:04.751 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:04.751 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:04.751 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:04.751 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:04.751 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:04.751 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.748+0000 7f402924a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:04.764 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.764+0000 7f402259c640 1 -- 172.21.15.66:0/4016251368 <== mon.2 v1:172.21.15.66:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f400c001960 con 0x7f402413f710 2024-03-25T18:06:04.764 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.764+0000 7f402259c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:04.764 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.764+0000 7f402259c640 1 -- 172.21.15.66:0/4016251368 <== mon.2 v1:172.21.15.66:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f400c001cf0 con 0x7f402413f710 2024-03-25T18:06:04.764 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.764+0000 7f402259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:04.764 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.764+0000 7f402259c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:06:04.765 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.764+0000 7f402259c640 10 monclient: dump: 2024-03-25T18:06:04.765 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:04.765 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:04.765 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:04.765 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:04.765 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:04.765 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:04.765 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:04.765 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:04.765 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:04.765 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:04.765 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.764+0000 7f402924a640 10 monclient: get_monmap_and_config success 2024-03-25T18:06:04.766 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.764+0000 7f402924a640 4 set_mon_vals no callback set 2024-03-25T18:06:04.768 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.764+0000 7f402924a640 10 monclient: shutdown 2024-03-25T18:06:04.768 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.764+0000 7f402924a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:04.768 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.764+0000 7f402924a640 1 -- 172.21.15.66:0/4016251368 >> v1:172.21.15.66:6790/0 conn(0x7f402413f710 legacy=0x7f40241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:04.768 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.764+0000 7f402924a640 1 -- 172.21.15.66:0/4016251368 shutdown_connections 2024-03-25T18:06:04.768 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.765+0000 7f402924a640 1 -- 172.21.15.66:0/4016251368 >> 172.21.15.66:0/4016251368 conn(0x7f40240b7ea0 msgr2=0x7f40240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:04.769 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.765+0000 7f402924a640 1 -- 172.21.15.66:0/4016251368 shutdown_connections 2024-03-25T18:06:04.769 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.765+0000 7f402924a640 1 -- 172.21.15.66:0/4016251368 wait complete. 2024-03-25T18:06:04.769 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.765+0000 7f402924a640 10 monclient: build_initial_monmap 2024-03-25T18:06:04.769 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.765+0000 7f402924a640 10 monclient: monmap: 2024-03-25T18:06:04.769 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:06:04.769 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:06:04.769 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:06:04.769 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:06:04.769 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:04.769 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:06:04.769 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:06:04.770 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:06:04.770 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:06:04.770 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:04.770 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.766+0000 7f402924a640 1 librados: starting msgr at 2024-03-25T18:06:04.770 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.766+0000 7f402924a640 1 librados: starting objecter 2024-03-25T18:06:04.770 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.766+0000 7f402924a640 1 Processor -- start 2024-03-25T18:06:04.770 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.766+0000 7f402924a640 1 -- start start 2024-03-25T18:06:04.770 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.766+0000 7f402924a640 1 librados: setting wanted keys 2024-03-25T18:06:04.770 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.766+0000 7f402924a640 1 librados: calling monclient init 2024-03-25T18:06:04.770 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.766+0000 7f402924a640 10 monclient: init 2024-03-25T18:06:04.770 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.767+0000 7f402924a640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:04.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.767+0000 7f402924a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:06:04.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.767+0000 7f402924a640 10 monclient(hunting): picked mon.noname-a con 0x7f40240a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:06:04.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.767+0000 7f402924a640 10 monclient(hunting): picked mon.noname-c con 0x7f402413f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:06:04.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.767+0000 7f402924a640 10 monclient(hunting): picked mon.noname-b con 0x7f40241430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:06:04.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.767+0000 7f402924a640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40241349e0 con 0x7f40240a98f0 2024-03-25T18:06:04.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.767+0000 7f402924a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f402406fd60 con 0x7f40241430c0 2024-03-25T18:06:04.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.767+0000 7f402924a640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f402414bee0 con 0x7f402413f710 2024-03-25T18:06:04.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.767+0000 7f402924a640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:04.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.767+0000 7f402924a640 10 monclient(hunting): authenticate will time out at 7113.826650s 2024-03-25T18:06:04.781 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.781+0000 7f401bfff640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f402413f710 0x7f4024133bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:45356/0 (socket says 172.21.15.66:45356) 2024-03-25T18:06:04.781 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.781+0000 7f401bfff640 1 -- 172.21.15.66:0/3483700415 learned_addr learned my addr 172.21.15.66:0/3483700415 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:06:04.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.784+0000 7f401b7fe640 10 client.?.objecter ms_handle_connect 0x7f40241430c0 2024-03-25T18:06:04.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.784+0000 7f401b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:04.785 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.785+0000 7f401b7fe640 10 client.?.objecter ms_handle_connect 0x7f402413f710 2024-03-25T18:06:04.785 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.785+0000 7f401b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:04.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.791+0000 7f401b7fe640 1 -- 172.21.15.66:0/3483700415 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1317795910 0 0) 0x7f402406fd60 con 0x7f40241430c0 2024-03-25T18:06:04.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.791+0000 7f401b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:06:04.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.791+0000 7f401b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:04.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.791+0000 7f401b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:04.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.791+0000 7f401b7fe640 10 monclient(hunting): my global_id is 4321 2024-03-25T18:06:04.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.791+0000 7f401b7fe640 1 -- 172.21.15.66:0/3483700415 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4004004eb0 con 0x7f40241430c0 2024-03-25T18:06:04.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.794+0000 7f401b7fe640 1 -- 172.21.15.66:0/3483700415 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 762675260 0 0) 0x7f402414bee0 con 0x7f402413f710 2024-03-25T18:06:04.795 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.794+0000 7f401b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:06:04.795 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.794+0000 7f401b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:04.795 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.794+0000 7f401b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:04.795 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.794+0000 7f401b7fe640 10 monclient(hunting): my global_id is 4322 2024-03-25T18:06:04.795 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.794+0000 7f401b7fe640 1 -- 172.21.15.66:0/3483700415 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f402406fd60 con 0x7f402413f710 2024-03-25T18:06:04.795 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.794+0000 7f401b7fe640 10 client.?.objecter ms_handle_connect 0x7f40240a98f0 2024-03-25T18:06:04.795 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.794+0000 7f401b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:04.803 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.803+0000 7f401b7fe640 1 -- 172.21.15.66:0/3483700415 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3747334566 0 0) 0x7f402406fd60 con 0x7f402413f710 2024-03-25T18:06:04.803 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.803+0000 7f401b7fe640 1 -- 172.21.15.66:0/3483700415 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f402414bee0 con 0x7f402413f710 2024-03-25T18:06:04.804 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.803+0000 7f401b7fe640 1 -- 172.21.15.66:0/3483700415 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f400c001650 con 0x7f402413f710 2024-03-25T18:06:04.804 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.803+0000 7f401b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:04.804 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.803+0000 7f401b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:06:04.804 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.803+0000 7f401b7fe640 10 monclient(hunting): dump: 2024-03-25T18:06:04.804 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:04.804 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:04.804 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:04.804 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:04.804 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:04.804 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:04.804 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:04.805 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:04.805 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:04.805 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:04.805 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.803+0000 7f401b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:06:04.806 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.805+0000 7f401b7fe640 1 -- 172.21.15.66:0/3483700415 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1343950871 0 0) 0x7f4004004eb0 con 0x7f40241430c0 2024-03-25T18:06:04.806 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.805+0000 7f401b7fe640 1 -- 172.21.15.66:0/3483700415 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f402406fd60 con 0x7f40241430c0 2024-03-25T18:06:04.806 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.805+0000 7f401b7fe640 1 -- 172.21.15.66:0/3483700415 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f401c001b90 con 0x7f40241430c0 2024-03-25T18:06:04.806 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.805+0000 7f401b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:04.806 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.805+0000 7f401b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:04.806 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.805+0000 7f401b7fe640 10 monclient(hunting): dump: 2024-03-25T18:06:04.807 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:04.807 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:04.807 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:04.807 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:04.807 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:04.807 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:04.807 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:04.807 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:04.807 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:04.807 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:04.810 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.809+0000 7f401b7fe640 1 -- 172.21.15.66:0/3483700415 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2667873820 0 0) 0x7f40241349e0 con 0x7f40240a98f0 2024-03-25T18:06:04.810 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.809+0000 7f401b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:06:04.810 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.809+0000 7f401b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:04.810 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.809+0000 7f401b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:04.810 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.809+0000 7f401b7fe640 10 monclient(hunting): my global_id is 4323 2024-03-25T18:06:04.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.810+0000 7f401b7fe640 1 -- 172.21.15.66:0/3483700415 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4004004eb0 con 0x7f40240a98f0 2024-03-25T18:06:04.825 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.823+0000 7f401b7fe640 1 -- 172.21.15.66:0/3483700415 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1147469003 0 0) 0x7f402406fd60 con 0x7f40241430c0 2024-03-25T18:06:04.825 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.823+0000 7f401b7fe640 1 -- 172.21.15.66:0/3483700415 >> v1:172.21.15.66:6790/0 conn(0x7f402413f710 legacy=0x7f4024133bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:04.825 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.823+0000 7f401b7fe640 1 -- 172.21.15.66:0/3483700415 >> v1:172.21.15.66:6789/0 conn(0x7f40240a98f0 legacy=0x7f40241dbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:04.825 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.823+0000 7f401b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:06:04.825 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.823+0000 7f401b7fe640 1 monclient: found mon.b 2024-03-25T18:06:04.825 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.823+0000 7f401b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:04.825 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.823+0000 7f401b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:06:04.825 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.823+0000 7f401b7fe640 1 -- 172.21.15.66:0/3483700415 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f402414c070 con 0x7f40241430c0 2024-03-25T18:06:04.825 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.823+0000 7f401b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:06:04.826 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.823+0000 7f401b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:06:04.826 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.823+0000 7f401b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:04.826 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.823+0000 7f402924a640 5 monclient: authenticate success, global_id 4321 2024-03-25T18:06:04.826 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.823+0000 7f402924a640 10 monclient: _renew_subs 2024-03-25T18:06:04.826 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.823+0000 7f402924a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:06:04.826 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.823+0000 7f402924a640 1 -- 172.21.15.66:0/3483700415 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40241dc340 con 0x7f40241430c0 2024-03-25T18:06:04.826 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.823+0000 7f402924a640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:06:04.826 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.823+0000 7f402924a640 10 monclient: _renew_subs 2024-03-25T18:06:04.826 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.823+0000 7f402924a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:06:04.826 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.823+0000 7f402924a640 1 -- 172.21.15.66:0/3483700415 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f402406e080 con 0x7f40241430c0 2024-03-25T18:06:04.826 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.823+0000 7f402924a640 1 librados: init done 2024-03-25T18:06:04.827 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.824+0000 7f402924a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:04.827 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.824+0000 7f402924a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:04.827 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.824+0000 7f402924a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:06:04.827 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:04.824+0000 7f402924a640 1 -- 172.21.15.66:0/3483700415 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40240b3ef0 con 0x7f40241430c0 2024-03-25T18:06:05.107 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.106+0000 7f401b7fe640 1 -- 172.21.15.66:0/3483700415 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f401c002060 con 0x7f40241430c0 2024-03-25T18:06:05.108 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.106+0000 7f401b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:05.108 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.106+0000 7f401b7fe640 1 -- 172.21.15.66:0/3483700415 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f401c003b10 con 0x7f40241430c0 2024-03-25T18:06:05.108 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.106+0000 7f401b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:05.108 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.106+0000 7f401b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:05.108 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.106+0000 7f401b7fe640 10 monclient: dump: 2024-03-25T18:06:05.108 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:05.108 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:05.108 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:05.109 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:05.109 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:05.109 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:05.109 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:05.109 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:05.109 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:05.109 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:05.109 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.106+0000 7f4021d9b640 4 set_mon_vals no callback set 2024-03-25T18:06:05.109 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.107+0000 7f401b7fe640 1 -- 172.21.15.66:0/3483700415 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f401c003d30 con 0x7f40241430c0 2024-03-25T18:06:05.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.108+0000 7f4022d9d640 10 monclient: get_auth_request con 0x7f400406dee0 auth_method 0 2024-03-25T18:06:05.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.108+0000 7f401b7fe640 10 client.4321.objecter ms_handle_connect 0x7f400406dee0 2024-03-25T18:06:05.117 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.117+0000 7f401b7fe640 1 -- 172.21.15.66:0/3483700415 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f401c0538b0 con 0x7f40241430c0 2024-03-25T18:06:05.118 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.117+0000 7f401b7fe640 10 client.4321.objecter ms_dispatch 0x7f4024000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:06:05.118 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.117+0000 7f401b7fe640 3 client.4321.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:06:05.118 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.117+0000 7f401b7fe640 3 client.4321.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:06:05.118 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.117+0000 7f401b7fe640 20 client.4321.objecter dump_active .. 0 homeless 2024-03-25T18:06:05.118 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.118+0000 7f401b7fe640 1 -- 172.21.15.66:0/3483700415 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f401c04f3c0 con 0x7f40241430c0 2024-03-25T18:06:05.119 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.118+0000 7f401b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:05.119 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.118+0000 7f401b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:06:05.370 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.370+0000 7f402924a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:05.371 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.370+0000 7f402924a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:05.371 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.370+0000 7f402924a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:06:05.371 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.370+0000 7f402924a640 1 -- 172.21.15.66:0/3483700415 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40240627b0 con 0x7f40241430c0 2024-03-25T18:06:05.399 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.398+0000 7f401b7fe640 1 -- 172.21.15.66:0/3483700415 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f401c0512f0 con 0x7f40241430c0 2024-03-25T18:06:05.399 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.398+0000 7f401b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:05.399 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.398+0000 7f401b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:06:05.399 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:06:05.399 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:06:05.402 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.401+0000 7f402924a640 10 librados: watch_flush enter 2024-03-25T18:06:05.402 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.401+0000 7f402924a640 10 librados: watch_flush exit 2024-03-25T18:06:05.402 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.401+0000 7f402924a640 20 client.4321.objecter shutdown clearing up homeless session... 2024-03-25T18:06:05.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.401+0000 7f402924a640 10 client.4321.objecter successfully canceled tick 2024-03-25T18:06:05.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.401+0000 7f402924a640 1 -- 172.21.15.66:0/3483700415 >> v1:172.21.15.193:6800/2275676803 conn(0x7f400406dee0 legacy=0x7f40040703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:05.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.401+0000 7f402924a640 10 monclient: shutdown 2024-03-25T18:06:05.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.401+0000 7f402924a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:05.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.401+0000 7f402924a640 1 -- 172.21.15.66:0/3483700415 >> v1:172.21.15.193:6789/0 conn(0x7f40241430c0 legacy=0x7f40241342d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:05.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.401+0000 7f402924a640 1 -- 172.21.15.66:0/3483700415 shutdown_connections 2024-03-25T18:06:05.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.401+0000 7f402924a640 1 -- 172.21.15.66:0/3483700415 >> 172.21.15.66:0/3483700415 conn(0x7f40240b7ea0 msgr2=0x7f4024074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:05.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.401+0000 7f402924a640 1 -- 172.21.15.66:0/3483700415 shutdown_connections 2024-03-25T18:06:05.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.401+0000 7f402924a640 1 -- 172.21.15.66:0/3483700415 wait complete. 2024-03-25T18:06:05.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:05.402+0000 7f402924a640 1 librados: shutdown 2024-03-25T18:06:05.415 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:06:12.221 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:06:12.340 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.340+0000 7f798788d640 10 monclient: get_monmap_and_config 2024-03-25T18:06:12.340 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.340+0000 7f798788d640 10 monclient: build_initial_monmap 2024-03-25T18:06:12.340 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.340+0000 7f798788d640 10 monclient: monmap: 2024-03-25T18:06:12.340 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:06:12.340 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:12.340 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:06:12.340764+0000 2024-03-25T18:06:12.340 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:06:12.340764+0000 2024-03-25T18:06:12.340 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:12.341 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:12.341 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:06:12.341 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:06:12.341 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:06:12.341 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:12.342 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.340+0000 7f798788d640 1 Processor -- start 2024-03-25T18:06:12.342 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.341+0000 7f798788d640 1 -- start start 2024-03-25T18:06:12.343 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.341+0000 7f798788d640 10 monclient: init 2024-03-25T18:06:12.343 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.341+0000 7f798788d640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:12.343 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.341+0000 7f798788d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:06:12.343 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.341+0000 7f798788d640 10 monclient(hunting): picked mon.noname-b con 0x7f79800b4d60 addr v1:172.21.15.66:6790/0 2024-03-25T18:06:12.343 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.341+0000 7f798788d640 10 monclient(hunting): picked mon.noname-c con 0x7f79800b1730 addr v1:172.21.15.193:6789/0 2024-03-25T18:06:12.343 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.341+0000 7f798788d640 10 monclient(hunting): picked mon.noname-a con 0x7f798014be90 addr v1:172.21.15.66:6789/0 2024-03-25T18:06:12.343 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.341+0000 7f798788d640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79800af940 con 0x7f798014be90 2024-03-25T18:06:12.343 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.341+0000 7f798788d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79800b0a70 con 0x7f79800b1730 2024-03-25T18:06:12.343 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.341+0000 7f798788d640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79800b0d50 con 0x7f79800b4d60 2024-03-25T18:06:12.343 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.341+0000 7f798788d640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:12.343 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.341+0000 7f798788d640 10 monclient(hunting): authenticate will time out at 7121.400851s 2024-03-25T18:06:12.343 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.342+0000 7f7985629640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f79800b4d60 0x7f798013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:50018/0 (socket says 172.21.15.66:50018) 2024-03-25T18:06:12.343 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.342+0000 7f7985629640 1 -- 172.21.15.66:0/1271579759 learned_addr learned my addr 172.21.15.66:0/1271579759 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:06:12.421 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.420+0000 7f7977fff640 1 -- 172.21.15.66:0/1271579759 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1387819980 0 0) 0x7f79800af940 con 0x7f798014be90 2024-03-25T18:06:12.421 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.420+0000 7f7977fff640 10 monclient(hunting): _negotiate 2024-03-25T18:06:12.421 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.420+0000 7f7977fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:12.421 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.420+0000 7f7977fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:12.421 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.420+0000 7f7977fff640 10 monclient(hunting): my global_id is 4326 2024-03-25T18:06:12.421 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.420+0000 7f7977fff640 1 -- 172.21.15.66:0/1271579759 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f796c003520 con 0x7f798014be90 2024-03-25T18:06:12.436 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.436+0000 7f7977fff640 1 -- 172.21.15.66:0/1271579759 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 426964035 0 0) 0x7f796c003520 con 0x7f798014be90 2024-03-25T18:06:12.436 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.436+0000 7f7977fff640 1 -- 172.21.15.66:0/1271579759 >> v1:172.21.15.66:6790/0 conn(0x7f79800b4d60 legacy=0x7f798013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:12.436 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.436+0000 7f7977fff640 1 -- 172.21.15.66:0/1271579759 >> v1:172.21.15.193:6789/0 conn(0x7f79800b1730 legacy=0x7f79800b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:12.436 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.436+0000 7f7977fff640 10 monclient: _finish_hunting 0 2024-03-25T18:06:12.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.436+0000 7f7977fff640 1 monclient: found mon.noname-a 2024-03-25T18:06:12.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.436+0000 7f7977fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:12.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.436+0000 7f7977fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.66:6789/0 2024-03-25T18:06:12.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.436+0000 7f7977fff640 1 -- 172.21.15.66:0/1271579759 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79800b1030 con 0x7f798014be90 2024-03-25T18:06:12.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.436+0000 7f7977fff640 10 monclient: _finish_auth 0 2024-03-25T18:06:12.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.436+0000 7f7977fff640 10 monclient: _check_auth_tickets 2024-03-25T18:06:12.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.436+0000 7f7977fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:12.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.436+0000 7f7977fff640 1 -- 172.21.15.66:0/1271579759 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7978001a10 con 0x7f798014be90 2024-03-25T18:06:12.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.436+0000 7f7977fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:12.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.436+0000 7f7977fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:06:12.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.436+0000 7f7977fff640 10 monclient: dump: 2024-03-25T18:06:12.437 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:12.437 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:12.437 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:12.437 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:12.438 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:12.438 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:12.438 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:12.438 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:12.438 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:12.438 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:12.438 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.436+0000 7f798788d640 5 monclient: authenticate success, global_id 4326 2024-03-25T18:06:12.438 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.436+0000 7f798788d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:12.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.446+0000 7f7977fff640 1 -- 172.21.15.66:0/1271579759 <== mon.0 v1:172.21.15.66:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7978001b70 con 0x7f798014be90 2024-03-25T18:06:12.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.446+0000 7f7977fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:12.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.446+0000 7f7977fff640 1 -- 172.21.15.66:0/1271579759 <== mon.0 v1:172.21.15.66:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7978001f80 con 0x7f798014be90 2024-03-25T18:06:12.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.446+0000 7f7977fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:12.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.446+0000 7f7977fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:06:12.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.446+0000 7f7977fff640 10 monclient: dump: 2024-03-25T18:06:12.447 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:12.447 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:12.447 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:12.447 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:12.447 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:12.447 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:12.448 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:12.448 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:12.448 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:12.448 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:12.448 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.446+0000 7f798788d640 10 monclient: get_monmap_and_config success 2024-03-25T18:06:12.448 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.446+0000 7f798788d640 4 set_mon_vals no callback set 2024-03-25T18:06:12.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.447+0000 7f798788d640 10 monclient: shutdown 2024-03-25T18:06:12.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.447+0000 7f798788d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:12.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.447+0000 7f798788d640 1 -- 172.21.15.66:0/1271579759 >> v1:172.21.15.66:6789/0 conn(0x7f798014be90 legacy=0x7f798014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:12.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.447+0000 7f798788d640 1 -- 172.21.15.66:0/1271579759 shutdown_connections 2024-03-25T18:06:12.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.447+0000 7f798788d640 1 -- 172.21.15.66:0/1271579759 >> 172.21.15.66:0/1271579759 conn(0x7f79800617a0 msgr2=0x7f7980061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:12.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.447+0000 7f798788d640 1 -- 172.21.15.66:0/1271579759 shutdown_connections 2024-03-25T18:06:12.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.447+0000 7f798788d640 1 -- 172.21.15.66:0/1271579759 wait complete. 2024-03-25T18:06:12.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.448+0000 7f798788d640 10 monclient: build_initial_monmap 2024-03-25T18:06:12.451 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.448+0000 7f798788d640 10 monclient: monmap: 2024-03-25T18:06:12.451 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:06:12.451 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:06:12.451 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:06:12.451 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:06:12.451 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:12.451 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:06:12.451 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:06:12.451 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:06:12.451 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:06:12.451 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:12.451 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.448+0000 7f798788d640 1 librados: starting msgr at 2024-03-25T18:06:12.451 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.448+0000 7f798788d640 1 librados: starting objecter 2024-03-25T18:06:12.451 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.448+0000 7f798788d640 1 Processor -- start 2024-03-25T18:06:12.451 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.449+0000 7f798788d640 1 -- start start 2024-03-25T18:06:12.451 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.449+0000 7f798788d640 1 librados: setting wanted keys 2024-03-25T18:06:12.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.449+0000 7f798788d640 1 librados: calling monclient init 2024-03-25T18:06:12.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.449+0000 7f798788d640 10 monclient: init 2024-03-25T18:06:12.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.449+0000 7f798788d640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:12.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.449+0000 7f798788d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:06:12.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.449+0000 7f798788d640 10 monclient(hunting): picked mon.noname-c con 0x7f79800b1730 addr v1:172.21.15.66:6790/0 2024-03-25T18:06:12.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.449+0000 7f798788d640 10 monclient(hunting): picked mon.noname-b con 0x7f79800b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T18:06:12.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.449+0000 7f798788d640 10 monclient(hunting): picked mon.noname-a con 0x7f798014be90 addr v1:172.21.15.66:6789/0 2024-03-25T18:06:12.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.449+0000 7f798788d640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7980140380 con 0x7f798014be90 2024-03-25T18:06:12.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.449+0000 7f798788d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7980140660 con 0x7f79800b4d60 2024-03-25T18:06:12.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.449+0000 7f798788d640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7980141e30 con 0x7f79800b1730 2024-03-25T18:06:12.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.449+0000 7f798788d640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:12.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.449+0000 7f798788d640 10 monclient(hunting): authenticate will time out at 7121.508807s 2024-03-25T18:06:12.462 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.462+0000 7f7985e2a640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f798014be90 0x7f79801e2740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:47744/0 (socket says 172.21.15.66:47744) 2024-03-25T18:06:12.462 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.462+0000 7f7985e2a640 1 -- 172.21.15.66:0/1445274787 learned_addr learned my addr 172.21.15.66:0/1445274787 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:06:12.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.465+0000 7f7975ffb640 10 client.?.objecter ms_handle_connect 0x7f798014be90 2024-03-25T18:06:12.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.465+0000 7f7975ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:12.466 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.466+0000 7f7975ffb640 10 client.?.objecter ms_handle_connect 0x7f79800b4d60 2024-03-25T18:06:12.466 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.466+0000 7f7975ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:12.474 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.474+0000 7f7975ffb640 1 -- 172.21.15.66:0/1445274787 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3394384887 0 0) 0x7f7980140660 con 0x7f79800b4d60 2024-03-25T18:06:12.474 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.474+0000 7f7975ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:06:12.474 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.474+0000 7f7975ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:12.474 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.474+0000 7f7975ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:12.474 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.474+0000 7f7975ffb640 10 monclient(hunting): my global_id is 4324 2024-03-25T18:06:12.475 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.474+0000 7f7975ffb640 1 -- 172.21.15.66:0/1445274787 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7964004eb0 con 0x7f79800b4d60 2024-03-25T18:06:12.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.477+0000 7f7975ffb640 1 -- 172.21.15.66:0/1445274787 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1627697966 0 0) 0x7f7980140380 con 0x7f798014be90 2024-03-25T18:06:12.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.477+0000 7f7975ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:06:12.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.477+0000 7f7975ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:12.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.477+0000 7f7975ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:12.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.477+0000 7f7975ffb640 10 monclient(hunting): my global_id is 4329 2024-03-25T18:06:12.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.477+0000 7f7975ffb640 1 -- 172.21.15.66:0/1445274787 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7980140660 con 0x7f798014be90 2024-03-25T18:06:12.483 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.483+0000 7f7975ffb640 10 client.?.objecter ms_handle_connect 0x7f79800b1730 2024-03-25T18:06:12.483 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.483+0000 7f7975ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:12.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.491+0000 7f7975ffb640 1 -- 172.21.15.66:0/1445274787 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2638560509 0 0) 0x7f7964004eb0 con 0x7f79800b4d60 2024-03-25T18:06:12.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.491+0000 7f7975ffb640 1 -- 172.21.15.66:0/1445274787 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7980140380 con 0x7f79800b4d60 2024-03-25T18:06:12.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.491+0000 7f7975ffb640 1 -- 172.21.15.66:0/1445274787 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f79700018c0 con 0x7f79800b4d60 2024-03-25T18:06:12.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.491+0000 7f7975ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:12.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.491+0000 7f7975ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:06:12.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.491+0000 7f7975ffb640 10 monclient(hunting): dump: 2024-03-25T18:06:12.492 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:12.492 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:12.492 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:12.492 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:12.492 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:12.492 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:12.492 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:12.492 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:12.492 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:12.492 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:12.492 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.491+0000 7f7975ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:06:12.506 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.505+0000 7f7975ffb640 1 -- 172.21.15.66:0/1445274787 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3487274125 0 0) 0x7f7980140380 con 0x7f79800b4d60 2024-03-25T18:06:12.506 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.505+0000 7f7975ffb640 1 -- 172.21.15.66:0/1445274787 >> v1:172.21.15.66:6790/0 conn(0x7f79800b1730 legacy=0x7f7980138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:12.506 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.505+0000 7f7975ffb640 1 -- 172.21.15.66:0/1445274787 >> v1:172.21.15.66:6789/0 conn(0x7f798014be90 legacy=0x7f79801e2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:12.506 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.505+0000 7f7975ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:06:12.506 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.505+0000 7f7975ffb640 1 monclient: found mon.b 2024-03-25T18:06:12.506 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.505+0000 7f7975ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:12.506 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.505+0000 7f7975ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:06:12.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.505+0000 7f7975ffb640 1 -- 172.21.15.66:0/1445274787 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7980142110 con 0x7f79800b4d60 2024-03-25T18:06:12.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.505+0000 7f7975ffb640 10 monclient: _finish_auth 0 2024-03-25T18:06:12.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.505+0000 7f7975ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:06:12.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.505+0000 7f7975ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:12.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.505+0000 7f798788d640 5 monclient: authenticate success, global_id 4324 2024-03-25T18:06:12.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.505+0000 7f798788d640 10 monclient: _renew_subs 2024-03-25T18:06:12.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.505+0000 7f798788d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:06:12.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.505+0000 7f798788d640 1 -- 172.21.15.66:0/1445274787 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7980140e60 con 0x7f79800b4d60 2024-03-25T18:06:12.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.506+0000 7f798788d640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:06:12.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.506+0000 7f798788d640 10 monclient: _renew_subs 2024-03-25T18:06:12.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.506+0000 7f798788d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:06:12.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.506+0000 7f798788d640 1 -- 172.21.15.66:0/1445274787 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7980142c40 con 0x7f79800b4d60 2024-03-25T18:06:12.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.506+0000 7f798788d640 1 librados: init done 2024-03-25T18:06:12.508 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.507+0000 7f798788d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:12.508 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.507+0000 7f798788d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:12.508 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.507+0000 7f798788d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:06:12.508 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.507+0000 7f798788d640 1 -- 172.21.15.66:0/1445274787 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79800627b0 con 0x7f79800b4d60 2024-03-25T18:06:12.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.793+0000 7f7975ffb640 1 -- 172.21.15.66:0/1445274787 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79700035f0 con 0x7f79800b4d60 2024-03-25T18:06:12.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.793+0000 7f7975ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:12.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.793+0000 7f7975ffb640 1 -- 172.21.15.66:0/1445274787 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7970003a00 con 0x7f79800b4d60 2024-03-25T18:06:12.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.793+0000 7f7975ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:12.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.793+0000 7f7975ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:12.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.793+0000 7f7975ffb640 10 monclient: dump: 2024-03-25T18:06:12.795 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:12.795 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:12.795 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:12.795 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:12.795 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:12.795 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:12.795 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:12.795 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:12.795 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:12.795 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:12.795 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.793+0000 7f79777fe640 4 set_mon_vals no callback set 2024-03-25T18:06:12.796 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.794+0000 7f7975ffb640 1 -- 172.21.15.66:0/1445274787 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f7970003c20 con 0x7f79800b4d60 2024-03-25T18:06:12.796 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.795+0000 7f7985629640 10 monclient: get_auth_request con 0x7f796406dd10 auth_method 0 2024-03-25T18:06:12.796 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.795+0000 7f7975ffb640 10 client.4324.objecter ms_handle_connect 0x7f796406dd10 2024-03-25T18:06:12.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.799+0000 7f7975ffb640 1 -- 172.21.15.66:0/1445274787 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f79700535b0 con 0x7f79800b4d60 2024-03-25T18:06:12.800 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.799+0000 7f7975ffb640 10 client.4324.objecter ms_dispatch 0x7f7980000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:06:12.800 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.799+0000 7f7975ffb640 3 client.4324.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:06:12.800 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.799+0000 7f7975ffb640 3 client.4324.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:06:12.800 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.799+0000 7f7975ffb640 20 client.4324.objecter dump_active .. 0 homeless 2024-03-25T18:06:12.800 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.799+0000 7f7975ffb640 1 -- 172.21.15.66:0/1445274787 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f797004f0c0 con 0x7f79800b4d60 2024-03-25T18:06:12.800 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.799+0000 7f7975ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:12.800 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:12.799+0000 7f7975ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:06:13.057 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:13.057+0000 7f798788d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:13.057 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:13.057+0000 7f798788d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:13.057 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:13.057+0000 7f798788d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:06:13.057 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:13.057+0000 7f798788d640 1 -- 172.21.15.66:0/1445274787 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7980140810 con 0x7f79800b4d60 2024-03-25T18:06:13.086 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:13.086+0000 7f7975ffb640 1 -- 172.21.15.66:0/1445274787 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f7980140810 con 0x7f79800b4d60 2024-03-25T18:06:13.086 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:13.086+0000 7f7975ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:13.086 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:13.086+0000 7f7975ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:06:13.086 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:06:13.087 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:06:13.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:13.088+0000 7f798788d640 10 librados: watch_flush enter 2024-03-25T18:06:13.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:13.088+0000 7f798788d640 10 librados: watch_flush exit 2024-03-25T18:06:13.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:13.088+0000 7f798788d640 20 client.4324.objecter shutdown clearing up homeless session... 2024-03-25T18:06:13.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:13.088+0000 7f798788d640 10 client.4324.objecter successfully canceled tick 2024-03-25T18:06:13.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:13.088+0000 7f798788d640 1 -- 172.21.15.66:0/1445274787 >> v1:172.21.15.193:6800/2275676803 conn(0x7f796406dd10 legacy=0x7f79640701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:13.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:13.088+0000 7f798788d640 10 monclient: shutdown 2024-03-25T18:06:13.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:13.088+0000 7f798788d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:13.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:13.088+0000 7f798788d640 1 -- 172.21.15.66:0/1445274787 >> v1:172.21.15.193:6789/0 conn(0x7f79800b4d60 legacy=0x7f7980142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:13.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:13.088+0000 7f798788d640 1 -- 172.21.15.66:0/1445274787 shutdown_connections 2024-03-25T18:06:13.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:13.088+0000 7f798788d640 1 -- 172.21.15.66:0/1445274787 >> 172.21.15.66:0/1445274787 conn(0x7f79800617a0 msgr2=0x7f79800b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:13.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:13.088+0000 7f798788d640 1 -- 172.21.15.66:0/1445274787 shutdown_connections 2024-03-25T18:06:13.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:13.088+0000 7f798788d640 1 -- 172.21.15.66:0/1445274787 wait complete. 2024-03-25T18:06:13.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:13.088+0000 7f798788d640 1 librados: shutdown 2024-03-25T18:06:13.099 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:06:19.906 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:06:20.020 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.020+0000 7f5ee3a6b640 10 monclient: get_monmap_and_config 2024-03-25T18:06:20.021 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.020+0000 7f5ee3a6b640 10 monclient: build_initial_monmap 2024-03-25T18:06:20.021 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.020+0000 7f5ee3a6b640 10 monclient: monmap: 2024-03-25T18:06:20.021 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:06:20.021 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:20.021 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:06:20.021162+0000 2024-03-25T18:06:20.021 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:06:20.021162+0000 2024-03-25T18:06:20.021 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:20.021 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:20.021 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:06:20.021 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:06:20.021 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:06:20.021 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:20.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.021+0000 7f5ee3a6b640 1 Processor -- start 2024-03-25T18:06:20.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.021+0000 7f5ee3a6b640 1 -- start start 2024-03-25T18:06:20.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.021+0000 7f5ee3a6b640 10 monclient: init 2024-03-25T18:06:20.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.022+0000 7f5ee3a6b640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:20.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.022+0000 7f5ee3a6b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:06:20.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.022+0000 7f5ee3a6b640 10 monclient(hunting): picked mon.noname-a con 0x7f5edc0a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:06:20.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.022+0000 7f5ee3a6b640 10 monclient(hunting): picked mon.noname-b con 0x7f5edc13f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:06:20.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.022+0000 7f5ee3a6b640 10 monclient(hunting): picked mon.noname-c con 0x7f5edc1430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:06:20.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.022+0000 7f5ee3a6b640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5edc142520 con 0x7f5edc0a98f0 2024-03-25T18:06:20.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.022+0000 7f5ee3a6b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5edc0b28a0 con 0x7f5edc1430c0 2024-03-25T18:06:20.024 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.022+0000 7f5ee3a6b640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5edc0b2a30 con 0x7f5edc13f710 2024-03-25T18:06:20.024 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.022+0000 7f5ee3a6b640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:20.024 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.022+0000 7f5ee3a6b640 10 monclient(hunting): authenticate will time out at 7129.081705s 2024-03-25T18:06:20.032 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.031+0000 7f5ee2008640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f5edc1430c0 0x7f5edc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:33738/0 (socket says 172.21.15.66:33738) 2024-03-25T18:06:20.032 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.032+0000 7f5ee2008640 1 -- 172.21.15.66:0/1020511896 learned_addr learned my addr 172.21.15.66:0/1020511896 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:06:20.039 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.039+0000 7f5ee0805640 1 -- 172.21.15.66:0/1020511896 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3334726299 0 0) 0x7f5edc0b28a0 con 0x7f5edc1430c0 2024-03-25T18:06:20.039 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.039+0000 7f5ee0805640 10 monclient(hunting): _negotiate 2024-03-25T18:06:20.039 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.039+0000 7f5ee0805640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:20.039 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.039+0000 7f5ee0805640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:20.039 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.039+0000 7f5ee0805640 10 monclient(hunting): my global_id is 4327 2024-03-25T18:06:20.039 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.039+0000 7f5ee0805640 1 -- 172.21.15.66:0/1020511896 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ec8003520 con 0x7f5edc1430c0 2024-03-25T18:06:20.048 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.047+0000 7f5ee0805640 1 -- 172.21.15.66:0/1020511896 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2227256720 0 0) 0x7f5ec8003520 con 0x7f5edc1430c0 2024-03-25T18:06:20.048 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.047+0000 7f5ee0805640 1 -- 172.21.15.66:0/1020511896 >> v1:172.21.15.66:6790/0 conn(0x7f5edc13f710 legacy=0x7f5edc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:20.048 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.047+0000 7f5ee0805640 1 -- 172.21.15.66:0/1020511896 >> v1:172.21.15.66:6789/0 conn(0x7f5edc0a98f0 legacy=0x7f5edc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:20.048 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.047+0000 7f5ee0805640 10 monclient: _finish_hunting 0 2024-03-25T18:06:20.048 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.047+0000 7f5ee0805640 1 monclient: found mon.noname-c 2024-03-25T18:06:20.048 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.047+0000 7f5ee0805640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:20.048 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.047+0000 7f5ee0805640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:06:20.048 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.047+0000 7f5ee0805640 1 -- 172.21.15.66:0/1020511896 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5edc0b44c0 con 0x7f5edc1430c0 2024-03-25T18:06:20.048 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.047+0000 7f5ee0805640 10 monclient: _finish_auth 0 2024-03-25T18:06:20.048 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.047+0000 7f5ee0805640 10 monclient: _check_auth_tickets 2024-03-25T18:06:20.048 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.047+0000 7f5ee0805640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:20.048 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.047+0000 7f5ee0805640 1 -- 172.21.15.66:0/1020511896 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f5ed4001a10 con 0x7f5edc1430c0 2024-03-25T18:06:20.048 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.047+0000 7f5ee0805640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:20.049 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.047+0000 7f5ee0805640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:06:20.049 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.047+0000 7f5ee0805640 10 monclient: dump: 2024-03-25T18:06:20.049 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:20.049 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:20.049 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:20.049 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:20.049 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:20.049 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:20.049 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:20.049 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:20.049 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:20.049 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:20.049 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.047+0000 7f5ee3a6b640 5 monclient: authenticate success, global_id 4327 2024-03-25T18:06:20.049 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.047+0000 7f5ee3a6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:20.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.064+0000 7f5ee0805640 1 -- 172.21.15.66:0/1020511896 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ed4001b70 con 0x7f5edc1430c0 2024-03-25T18:06:20.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.064+0000 7f5ee0805640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:20.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.064+0000 7f5ee0805640 1 -- 172.21.15.66:0/1020511896 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f5ed4001f80 con 0x7f5edc1430c0 2024-03-25T18:06:20.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.064+0000 7f5ee0805640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:20.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.064+0000 7f5ee0805640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:20.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.064+0000 7f5ee0805640 10 monclient: dump: 2024-03-25T18:06:20.065 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:20.065 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:20.065 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:20.065 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:20.065 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:20.065 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:20.065 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:20.065 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:20.065 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:20.065 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:20.066 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.064+0000 7f5ee3a6b640 10 monclient: get_monmap_and_config success 2024-03-25T18:06:20.067 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.064+0000 7f5ee3a6b640 4 set_mon_vals no callback set 2024-03-25T18:06:20.067 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.065+0000 7f5ee3a6b640 10 monclient: shutdown 2024-03-25T18:06:20.067 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.065+0000 7f5ee3a6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:20.067 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.065+0000 7f5ee3a6b640 1 -- 172.21.15.66:0/1020511896 >> v1:172.21.15.193:6789/0 conn(0x7f5edc1430c0 legacy=0x7f5edc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:20.067 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.065+0000 7f5ee3a6b640 1 -- 172.21.15.66:0/1020511896 shutdown_connections 2024-03-25T18:06:20.067 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.065+0000 7f5ee3a6b640 1 -- 172.21.15.66:0/1020511896 >> 172.21.15.66:0/1020511896 conn(0x7f5edc0b7ea0 msgr2=0x7f5edc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:20.067 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.065+0000 7f5ee3a6b640 1 -- 172.21.15.66:0/1020511896 shutdown_connections 2024-03-25T18:06:20.067 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.065+0000 7f5ee3a6b640 1 -- 172.21.15.66:0/1020511896 wait complete. 2024-03-25T18:06:20.067 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.066+0000 7f5ee3a6b640 10 monclient: build_initial_monmap 2024-03-25T18:06:20.067 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.066+0000 7f5ee3a6b640 10 monclient: monmap: 2024-03-25T18:06:20.067 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:06:20.067 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:06:20.067 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:06:20.067 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:06:20.067 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:20.068 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:06:20.068 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:06:20.068 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:06:20.068 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:06:20.068 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:20.069 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.067+0000 7f5ee3a6b640 1 librados: starting msgr at 2024-03-25T18:06:20.069 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.067+0000 7f5ee3a6b640 1 librados: starting objecter 2024-03-25T18:06:20.069 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.067+0000 7f5ee3a6b640 1 Processor -- start 2024-03-25T18:06:20.069 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.067+0000 7f5ee3a6b640 1 -- start start 2024-03-25T18:06:20.069 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.067+0000 7f5ee3a6b640 1 librados: setting wanted keys 2024-03-25T18:06:20.069 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.067+0000 7f5ee3a6b640 1 librados: calling monclient init 2024-03-25T18:06:20.069 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.067+0000 7f5ee3a6b640 10 monclient: init 2024-03-25T18:06:20.070 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.068+0000 7f5ee3a6b640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:20.070 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.068+0000 7f5ee3a6b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:06:20.070 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.068+0000 7f5ee3a6b640 10 monclient(hunting): picked mon.noname-c con 0x7f5edc0a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:06:20.070 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.068+0000 7f5ee3a6b640 10 monclient(hunting): picked mon.noname-b con 0x7f5edc13f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:06:20.070 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.068+0000 7f5ee3a6b640 10 monclient(hunting): picked mon.noname-a con 0x7f5edc1430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:06:20.070 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.068+0000 7f5ee3a6b640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5edc136ad0 con 0x7f5edc1430c0 2024-03-25T18:06:20.070 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.068+0000 7f5ee3a6b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5edc0b7520 con 0x7f5edc13f710 2024-03-25T18:06:20.070 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.068+0000 7f5ee3a6b640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5edc135810 con 0x7f5edc0a98f0 2024-03-25T18:06:20.070 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.068+0000 7f5ee3a6b640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:20.070 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.068+0000 7f5ee3a6b640 10 monclient(hunting): authenticate will time out at 7129.127253s 2024-03-25T18:06:20.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.088+0000 7f5ee1006640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f5edc13f710 0x7f5edc0b5a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:33794/0 (socket says 172.21.15.66:33794) 2024-03-25T18:06:20.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.089+0000 7f5ee1006640 1 -- 172.21.15.66:0/912446523 learned_addr learned my addr 172.21.15.66:0/912446523 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:06:20.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.089+0000 7f5ed27fc640 10 client.?.objecter ms_handle_connect 0x7f5edc13f710 2024-03-25T18:06:20.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.089+0000 7f5ed27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:20.093 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.093+0000 7f5ed27fc640 10 client.?.objecter ms_handle_connect 0x7f5edc1430c0 2024-03-25T18:06:20.093 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.093+0000 7f5ed27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:20.096 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.095+0000 7f5ed27fc640 1 -- 172.21.15.66:0/912446523 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2918608056 0 0) 0x7f5edc0b7520 con 0x7f5edc13f710 2024-03-25T18:06:20.096 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.095+0000 7f5ed27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:06:20.096 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.095+0000 7f5ed27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:20.096 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.095+0000 7f5ed27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:20.096 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.095+0000 7f5ed27fc640 10 monclient(hunting): my global_id is 4330 2024-03-25T18:06:20.096 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.095+0000 7f5ed27fc640 1 -- 172.21.15.66:0/912446523 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ec0004eb0 con 0x7f5edc13f710 2024-03-25T18:06:20.097 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.097+0000 7f5ed27fc640 10 client.?.objecter ms_handle_connect 0x7f5edc0a98f0 2024-03-25T18:06:20.097 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.097+0000 7f5ed27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:20.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.109+0000 7f5ed27fc640 1 -- 172.21.15.66:0/912446523 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2966240636 0 0) 0x7f5edc135810 con 0x7f5edc0a98f0 2024-03-25T18:06:20.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.109+0000 7f5ed27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:06:20.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.109+0000 7f5ed27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:20.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.109+0000 7f5ed27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:20.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.109+0000 7f5ed27fc640 10 monclient(hunting): my global_id is 4334 2024-03-25T18:06:20.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.109+0000 7f5ed27fc640 1 -- 172.21.15.66:0/912446523 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5edc0b7520 con 0x7f5edc0a98f0 2024-03-25T18:06:20.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.110+0000 7f5ed27fc640 1 -- 172.21.15.66:0/912446523 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2747736061 0 0) 0x7f5edc136ad0 con 0x7f5edc1430c0 2024-03-25T18:06:20.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.110+0000 7f5ed27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:06:20.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.110+0000 7f5ed27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:20.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.110+0000 7f5ed27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:20.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.110+0000 7f5ed27fc640 10 monclient(hunting): my global_id is 4335 2024-03-25T18:06:20.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.110+0000 7f5ed27fc640 1 -- 172.21.15.66:0/912446523 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5edc135810 con 0x7f5edc1430c0 2024-03-25T18:06:20.118 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.117+0000 7f5ed27fc640 1 -- 172.21.15.66:0/912446523 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2756288574 0 0) 0x7f5ec0004eb0 con 0x7f5edc13f710 2024-03-25T18:06:20.118 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.117+0000 7f5ed27fc640 1 -- 172.21.15.66:0/912446523 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5edc136ad0 con 0x7f5edc13f710 2024-03-25T18:06:20.118 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.117+0000 7f5ed27fc640 1 -- 172.21.15.66:0/912446523 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f5ecc001a30 con 0x7f5edc13f710 2024-03-25T18:06:20.118 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.117+0000 7f5ed27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:20.118 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.117+0000 7f5ed27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:06:20.118 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.117+0000 7f5ed27fc640 10 monclient(hunting): dump: 2024-03-25T18:06:20.118 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:20.118 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:20.118 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:20.119 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:20.119 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:20.119 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:20.119 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:20.119 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:20.119 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:20.119 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:20.119 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.117+0000 7f5ed27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:06:20.127 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.126+0000 7f5ed27fc640 1 -- 172.21.15.66:0/912446523 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1978595527 0 0) 0x7f5edc136ad0 con 0x7f5edc13f710 2024-03-25T18:06:20.127 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.126+0000 7f5ed27fc640 1 -- 172.21.15.66:0/912446523 >> v1:172.21.15.66:6790/0 conn(0x7f5edc0a98f0 legacy=0x7f5edc135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:20.127 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.126+0000 7f5ed27fc640 1 -- 172.21.15.66:0/912446523 >> v1:172.21.15.66:6789/0 conn(0x7f5edc1430c0 legacy=0x7f5edc130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:20.128 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.126+0000 7f5ed27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:06:20.128 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.126+0000 7f5ed27fc640 1 monclient: found mon.b 2024-03-25T18:06:20.128 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.126+0000 7f5ed27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:20.128 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.126+0000 7f5ed27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:06:20.128 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.126+0000 7f5ed27fc640 1 -- 172.21.15.66:0/912446523 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5edc0b6180 con 0x7f5edc13f710 2024-03-25T18:06:20.128 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.126+0000 7f5ed27fc640 10 monclient: _finish_auth 0 2024-03-25T18:06:20.128 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.126+0000 7f5ed27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:06:20.128 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.126+0000 7f5ed27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:20.128 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.126+0000 7f5ee3a6b640 5 monclient: authenticate success, global_id 4330 2024-03-25T18:06:20.128 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.126+0000 7f5ee3a6b640 10 monclient: _renew_subs 2024-03-25T18:06:20.128 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.126+0000 7f5ee3a6b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:06:20.128 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.126+0000 7f5ee3a6b640 1 -- 172.21.15.66:0/912446523 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5edc0b7a40 con 0x7f5edc13f710 2024-03-25T18:06:20.128 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.126+0000 7f5ee3a6b640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:06:20.129 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.126+0000 7f5ee3a6b640 10 monclient: _renew_subs 2024-03-25T18:06:20.129 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.126+0000 7f5ee3a6b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:06:20.129 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.126+0000 7f5ee3a6b640 1 -- 172.21.15.66:0/912446523 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5edc131840 con 0x7f5edc13f710 2024-03-25T18:06:20.129 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.126+0000 7f5ee3a6b640 1 librados: init done 2024-03-25T18:06:20.129 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.128+0000 7f5ee3a6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:20.129 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.128+0000 7f5ee3a6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:20.130 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.128+0000 7f5ee3a6b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:06:20.130 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.128+0000 7f5ee3a6b640 1 -- 172.21.15.66:0/912446523 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5edc136f60 con 0x7f5edc13f710 2024-03-25T18:06:20.420 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.419+0000 7f5ed27fc640 1 -- 172.21.15.66:0/912446523 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ecc001c70 con 0x7f5edc13f710 2024-03-25T18:06:20.420 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.419+0000 7f5ed27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:20.420 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.419+0000 7f5ed27fc640 1 -- 172.21.15.66:0/912446523 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f5ecc0038c0 con 0x7f5edc13f710 2024-03-25T18:06:20.420 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.419+0000 7f5ed27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:20.420 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.419+0000 7f5ed27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:20.420 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.419+0000 7f5ed27fc640 10 monclient: dump: 2024-03-25T18:06:20.420 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:20.420 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:20.420 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:20.421 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:20.421 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:20.421 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:20.421 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:20.421 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:20.421 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:20.421 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:20.421 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.419+0000 7f5ed3fff640 4 set_mon_vals no callback set 2024-03-25T18:06:20.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.420+0000 7f5ed27fc640 1 -- 172.21.15.66:0/912446523 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f5ecc003aa0 con 0x7f5edc13f710 2024-03-25T18:06:20.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.421+0000 7f5ee1807640 10 monclient: get_auth_request con 0x7f5ec006dda0 auth_method 0 2024-03-25T18:06:20.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.421+0000 7f5ed27fc640 10 client.4330.objecter ms_handle_connect 0x7f5ec006dda0 2024-03-25T18:06:20.425 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.424+0000 7f5ed27fc640 1 -- 172.21.15.66:0/912446523 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f5ecc054760 con 0x7f5edc13f710 2024-03-25T18:06:20.425 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.424+0000 7f5ed27fc640 10 client.4330.objecter ms_dispatch 0x7f5edc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:06:20.425 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.424+0000 7f5ed27fc640 3 client.4330.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:06:20.425 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.424+0000 7f5ed27fc640 3 client.4330.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:06:20.425 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.425+0000 7f5ed27fc640 20 client.4330.objecter dump_active .. 0 homeless 2024-03-25T18:06:20.425 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.425+0000 7f5ed27fc640 1 -- 172.21.15.66:0/912446523 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5ecc050270 con 0x7f5edc13f710 2024-03-25T18:06:20.426 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.425+0000 7f5ed27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:20.426 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.425+0000 7f5ed27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:06:20.660 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.660+0000 7f5ee3a6b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:20.660 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.660+0000 7f5ee3a6b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:20.660 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.660+0000 7f5ee3a6b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:06:20.660 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.660+0000 7f5ee3a6b640 1 -- 172.21.15.66:0/912446523 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5edc0b6310 con 0x7f5edc13f710 2024-03-25T18:06:20.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.695+0000 7f5ed27fc640 1 -- 172.21.15.66:0/912446523 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f5ecc0521a0 con 0x7f5edc13f710 2024-03-25T18:06:20.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.695+0000 7f5ed27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:20.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.695+0000 7f5ed27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:06:20.696 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:06:20.696 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:06:20.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.697+0000 7f5ee3a6b640 10 librados: watch_flush enter 2024-03-25T18:06:20.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.697+0000 7f5ee3a6b640 10 librados: watch_flush exit 2024-03-25T18:06:20.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.697+0000 7f5ee3a6b640 20 client.4330.objecter shutdown clearing up homeless session... 2024-03-25T18:06:20.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.697+0000 7f5ee3a6b640 10 client.4330.objecter successfully canceled tick 2024-03-25T18:06:20.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.698+0000 7f5ee3a6b640 1 -- 172.21.15.66:0/912446523 >> v1:172.21.15.193:6800/2275676803 conn(0x7f5ec006dda0 legacy=0x7f5ec0070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:20.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.698+0000 7f5ee3a6b640 10 monclient: shutdown 2024-03-25T18:06:20.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.698+0000 7f5ee3a6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:20.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.698+0000 7f5ee3a6b640 1 -- 172.21.15.66:0/912446523 >> v1:172.21.15.193:6789/0 conn(0x7f5edc13f710 legacy=0x7f5edc0b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:20.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.698+0000 7f5ee3a6b640 1 -- 172.21.15.66:0/912446523 shutdown_connections 2024-03-25T18:06:20.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.698+0000 7f5ee3a6b640 1 -- 172.21.15.66:0/912446523 >> 172.21.15.66:0/912446523 conn(0x7f5edc0b7ea0 msgr2=0x7f5edc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:20.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.698+0000 7f5ee3a6b640 1 -- 172.21.15.66:0/912446523 shutdown_connections 2024-03-25T18:06:20.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.698+0000 7f5ee3a6b640 1 -- 172.21.15.66:0/912446523 wait complete. 2024-03-25T18:06:20.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:20.698+0000 7f5ee3a6b640 1 librados: shutdown 2024-03-25T18:06:20.711 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:06:23.908 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:02:54.888 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:23.953 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:02:54.911 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:23.954 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:02:54.913 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:23.959 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:02:54.940 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:23.965 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:02:54.923 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:24.000 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:02:54.981 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:24.029 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:02:55.010 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:24.036 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:02:55.016 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:24.064 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:02:55.044 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:27.517 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:06:27.637 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.637+0000 7f4105a74640 10 monclient: get_monmap_and_config 2024-03-25T18:06:27.637 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.637+0000 7f4105a74640 10 monclient: build_initial_monmap 2024-03-25T18:06:27.637 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.637+0000 7f4105a74640 10 monclient: monmap: 2024-03-25T18:06:27.637 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:06:27.637 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:27.637 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:06:27.637593+0000 2024-03-25T18:06:27.637 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:06:27.637593+0000 2024-03-25T18:06:27.637 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:27.637 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:27.637 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:06:27.638 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:06:27.638 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:06:27.638 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:27.639 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.638+0000 7f4105a74640 1 Processor -- start 2024-03-25T18:06:27.639 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.638+0000 7f4105a74640 1 -- start start 2024-03-25T18:06:27.639 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.638+0000 7f4105a74640 10 monclient: init 2024-03-25T18:06:27.639 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.638+0000 7f4105a74640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:27.639 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.638+0000 7f4105a74640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:06:27.639 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.638+0000 7f4105a74640 10 monclient(hunting): picked mon.noname-b con 0x7f41000a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:06:27.639 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.639+0000 7f4105a74640 10 monclient(hunting): picked mon.noname-a con 0x7f410013f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:06:27.640 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.639+0000 7f4105a74640 10 monclient(hunting): picked mon.noname-c con 0x7f41001430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:06:27.640 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.639+0000 7f4105a74640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4100142520 con 0x7f410013f710 2024-03-25T18:06:27.640 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.639+0000 7f4105a74640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41000b28a0 con 0x7f41001430c0 2024-03-25T18:06:27.640 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.639+0000 7f4105a74640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41000b2a30 con 0x7f41000a98f0 2024-03-25T18:06:27.640 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.639+0000 7f4105a74640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:27.640 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.639+0000 7f4105a74640 10 monclient(hunting): authenticate will time out at 7136.698239s 2024-03-25T18:06:27.646 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.646+0000 7f40ff7fe640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f41000a98f0 0x7f41000a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:58772/0 (socket says 172.21.15.66:58772) 2024-03-25T18:06:27.646 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.646+0000 7f40ff7fe640 1 -- 172.21.15.66:0/3506261432 learned_addr learned my addr 172.21.15.66:0/3506261432 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:06:27.659 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.659+0000 7f40fe7fc640 1 -- 172.21.15.66:0/3506261432 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 525231389 0 0) 0x7f41000b2a30 con 0x7f41000a98f0 2024-03-25T18:06:27.659 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.659+0000 7f40fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:06:27.659 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.659+0000 7f40fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:27.659 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.659+0000 7f40fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:27.659 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.659+0000 7f40fe7fc640 10 monclient(hunting): my global_id is 4337 2024-03-25T18:06:27.659 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.659+0000 7f40fe7fc640 1 -- 172.21.15.66:0/3506261432 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40e8003520 con 0x7f41000a98f0 2024-03-25T18:06:27.668 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.667+0000 7f40fe7fc640 1 -- 172.21.15.66:0/3506261432 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 758809075 0 0) 0x7f40e8003520 con 0x7f41000a98f0 2024-03-25T18:06:27.668 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.667+0000 7f40fe7fc640 1 -- 172.21.15.66:0/3506261432 >> v1:172.21.15.193:6789/0 conn(0x7f41001430c0 legacy=0x7f410014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:27.668 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.667+0000 7f40fe7fc640 1 -- 172.21.15.66:0/3506261432 >> v1:172.21.15.66:6789/0 conn(0x7f410013f710 legacy=0x7f41001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:27.668 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.667+0000 7f40fe7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:06:27.668 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.667+0000 7f40fe7fc640 1 monclient: found mon.noname-b 2024-03-25T18:06:27.668 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.667+0000 7f40fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:27.668 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.667+0000 7f40fe7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.66:6790/0 2024-03-25T18:06:27.668 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.667+0000 7f40fe7fc640 1 -- 172.21.15.66:0/3506261432 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41000b44c0 con 0x7f41000a98f0 2024-03-25T18:06:27.668 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.667+0000 7f40fe7fc640 10 monclient: _finish_auth 0 2024-03-25T18:06:27.668 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.667+0000 7f40fe7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:06:27.668 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.667+0000 7f40fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:27.669 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.667+0000 7f40fe7fc640 1 -- 172.21.15.66:0/3506261432 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f40f0001690 con 0x7f41000a98f0 2024-03-25T18:06:27.669 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.667+0000 7f40fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:27.669 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.667+0000 7f40fe7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:06:27.669 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.667+0000 7f40fe7fc640 10 monclient: dump: 2024-03-25T18:06:27.669 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:27.669 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:27.669 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:27.669 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:27.669 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:27.669 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:27.669 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:27.669 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:27.669 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:27.669 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:27.670 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.667+0000 7f4105a74640 5 monclient: authenticate success, global_id 4337 2024-03-25T18:06:27.670 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.667+0000 7f4105a74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:27.676 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.676+0000 7f40fe7fc640 1 -- 172.21.15.66:0/3506261432 <== mon.2 v1:172.21.15.66:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40f00017f0 con 0x7f41000a98f0 2024-03-25T18:06:27.677 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.676+0000 7f40fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:27.677 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.676+0000 7f40fe7fc640 1 -- 172.21.15.66:0/3506261432 <== mon.2 v1:172.21.15.66:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f40f0001c40 con 0x7f41000a98f0 2024-03-25T18:06:27.677 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.676+0000 7f4105a74640 10 monclient: get_monmap_and_config success 2024-03-25T18:06:27.677 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.676+0000 7f4105a74640 4 set_mon_vals no callback set 2024-03-25T18:06:27.677 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.676+0000 7f40fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:27.677 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.676+0000 7f40fe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:06:27.677 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.676+0000 7f40fe7fc640 10 monclient: dump: 2024-03-25T18:06:27.677 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:27.677 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:27.677 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:27.677 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:27.677 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:27.677 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:27.678 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:27.678 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:27.678 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:27.678 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:27.679 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.677+0000 7f4105a74640 10 monclient: shutdown 2024-03-25T18:06:27.679 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.677+0000 7f4105a74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:27.679 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.677+0000 7f4105a74640 1 -- 172.21.15.66:0/3506261432 >> v1:172.21.15.66:6790/0 conn(0x7f41000a98f0 legacy=0x7f41000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:27.679 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.677+0000 7f4105a74640 1 -- 172.21.15.66:0/3506261432 shutdown_connections 2024-03-25T18:06:27.679 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.677+0000 7f4105a74640 1 -- 172.21.15.66:0/3506261432 >> 172.21.15.66:0/3506261432 conn(0x7f41000b7ea0 msgr2=0x7f41000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:27.679 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.677+0000 7f4105a74640 1 -- 172.21.15.66:0/3506261432 shutdown_connections 2024-03-25T18:06:27.680 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.677+0000 7f4105a74640 1 -- 172.21.15.66:0/3506261432 wait complete. 2024-03-25T18:06:27.680 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.678+0000 7f4105a74640 10 monclient: build_initial_monmap 2024-03-25T18:06:27.680 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.678+0000 7f4105a74640 10 monclient: monmap: 2024-03-25T18:06:27.680 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:06:27.680 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:06:27.680 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:06:27.680 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:06:27.680 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:27.680 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:06:27.680 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:06:27.680 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:06:27.680 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:06:27.680 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:27.680 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.678+0000 7f4105a74640 1 librados: starting msgr at 2024-03-25T18:06:27.680 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.678+0000 7f4105a74640 1 librados: starting objecter 2024-03-25T18:06:27.681 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.678+0000 7f4105a74640 1 Processor -- start 2024-03-25T18:06:27.681 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.679+0000 7f4105a74640 1 -- start start 2024-03-25T18:06:27.681 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.679+0000 7f4105a74640 1 librados: setting wanted keys 2024-03-25T18:06:27.681 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.679+0000 7f4105a74640 1 librados: calling monclient init 2024-03-25T18:06:27.681 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.679+0000 7f4105a74640 10 monclient: init 2024-03-25T18:06:27.682 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.680+0000 7f4105a74640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:27.682 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.680+0000 7f4105a74640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:06:27.682 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.680+0000 7f4105a74640 10 monclient(hunting): picked mon.noname-c con 0x7f41000a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:06:27.682 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.680+0000 7f4105a74640 10 monclient(hunting): picked mon.noname-a con 0x7f410013f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:06:27.682 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.680+0000 7f4105a74640 10 monclient(hunting): picked mon.noname-b con 0x7f41001430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:06:27.682 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.680+0000 7f4105a74640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4100135590 con 0x7f410013f710 2024-03-25T18:06:27.682 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.680+0000 7f4105a74640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4100136e50 con 0x7f41001430c0 2024-03-25T18:06:27.682 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.680+0000 7f4105a74640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4100135070 con 0x7f41000a98f0 2024-03-25T18:06:27.682 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.680+0000 7f4105a74640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:27.682 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.680+0000 7f4105a74640 10 monclient(hunting): authenticate will time out at 7136.739526s 2024-03-25T18:06:27.687 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.687+0000 7f40fffff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f41001430c0 0x7f41001e3010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:40918/0 (socket says 172.21.15.66:40918) 2024-03-25T18:06:27.687 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.687+0000 7f40fffff640 1 -- 172.21.15.66:0/1956436670 learned_addr learned my addr 172.21.15.66:0/1956436670 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:06:27.691 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.690+0000 7f40e7fff640 10 client.?.objecter ms_handle_connect 0x7f41001430c0 2024-03-25T18:06:27.691 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.690+0000 7f40e7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:27.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.695+0000 7f40e7fff640 10 client.?.objecter ms_handle_connect 0x7f41000a98f0 2024-03-25T18:06:27.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.695+0000 7f40e7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:27.697 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.697+0000 7f40e7fff640 10 client.?.objecter ms_handle_connect 0x7f410013f710 2024-03-25T18:06:27.697 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.697+0000 7f40e7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:27.706 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.706+0000 7f40e7fff640 1 -- 172.21.15.66:0/1956436670 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3868955247 0 0) 0x7f4100135070 con 0x7f41000a98f0 2024-03-25T18:06:27.706 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.706+0000 7f40e7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:06:27.706 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.706+0000 7f40e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:27.706 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.706+0000 7f40e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:27.706 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.706+0000 7f40e7fff640 10 monclient(hunting): my global_id is 4340 2024-03-25T18:06:27.707 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.706+0000 7f40e7fff640 1 -- 172.21.15.66:0/1956436670 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40ec004eb0 con 0x7f41000a98f0 2024-03-25T18:06:27.712 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.712+0000 7f40e7fff640 1 -- 172.21.15.66:0/1956436670 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 910996933 0 0) 0x7f4100136e50 con 0x7f41001430c0 2024-03-25T18:06:27.712 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.712+0000 7f40e7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:06:27.712 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.712+0000 7f40e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:27.712 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.712+0000 7f40e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:27.712 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.712+0000 7f40e7fff640 10 monclient(hunting): my global_id is 4336 2024-03-25T18:06:27.712 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.712+0000 7f40e7fff640 1 -- 172.21.15.66:0/1956436670 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4100135070 con 0x7f41001430c0 2024-03-25T18:06:27.727 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.727+0000 7f40e7fff640 1 -- 172.21.15.66:0/1956436670 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2906425098 0 0) 0x7f4100135590 con 0x7f410013f710 2024-03-25T18:06:27.727 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.727+0000 7f40e7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:06:27.727 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.727+0000 7f40e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:27.727 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.727+0000 7f40e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:27.728 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.727+0000 7f40e7fff640 10 monclient(hunting): my global_id is 4341 2024-03-25T18:06:27.728 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.727+0000 7f40e7fff640 1 -- 172.21.15.66:0/1956436670 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4100136e50 con 0x7f410013f710 2024-03-25T18:06:27.730 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.730+0000 7f40e7fff640 1 -- 172.21.15.66:0/1956436670 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 86566323 0 0) 0x7f40ec004eb0 con 0x7f41000a98f0 2024-03-25T18:06:27.730 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.730+0000 7f40e7fff640 1 -- 172.21.15.66:0/1956436670 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4100135590 con 0x7f41000a98f0 2024-03-25T18:06:27.730 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.730+0000 7f40e7fff640 1 -- 172.21.15.66:0/1956436670 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f40f0001b00 con 0x7f41000a98f0 2024-03-25T18:06:27.730 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.730+0000 7f40e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:27.730 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.730+0000 7f40e7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:06:27.731 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.730+0000 7f40e7fff640 10 monclient(hunting): dump: 2024-03-25T18:06:27.731 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:27.731 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:27.731 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:27.731 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:27.731 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:27.731 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:27.731 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:27.731 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:27.731 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:27.731 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:27.731 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.730+0000 7f40e7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:06:27.734 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.734+0000 7f40e7fff640 1 -- 172.21.15.66:0/1956436670 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3484010618 0 0) 0x7f4100135070 con 0x7f41001430c0 2024-03-25T18:06:27.734 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.734+0000 7f40e7fff640 1 -- 172.21.15.66:0/1956436670 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40ec004eb0 con 0x7f41001430c0 2024-03-25T18:06:27.734 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.734+0000 7f40e7fff640 1 -- 172.21.15.66:0/1956436670 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f40f8001bd0 con 0x7f41001430c0 2024-03-25T18:06:27.734 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.734+0000 7f40e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:27.734 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.734+0000 7f40e7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:27.734 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.734+0000 7f40e7fff640 10 monclient(hunting): dump: 2024-03-25T18:06:27.735 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:27.735 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:27.735 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:27.735 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:27.735 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:27.735 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:27.735 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:27.735 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:27.735 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:27.735 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:27.753 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.753+0000 7f40e7fff640 1 -- 172.21.15.66:0/1956436670 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3800278762 0 0) 0x7f4100136e50 con 0x7f410013f710 2024-03-25T18:06:27.753 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.753+0000 7f40e7fff640 1 -- 172.21.15.66:0/1956436670 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4100135070 con 0x7f410013f710 2024-03-25T18:06:27.753 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.753+0000 7f40e7fff640 1 -- 172.21.15.66:0/1956436670 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f40f4001830 con 0x7f410013f710 2024-03-25T18:06:27.753 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.753+0000 7f40e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:27.753 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.753+0000 7f40e7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:06:27.753 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.753+0000 7f40e7fff640 10 monclient(hunting): dump: 2024-03-25T18:06:27.753 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:27.753 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:27.753 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:27.753 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:27.753 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:27.754 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:27.754 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:27.754 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:27.754 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:27.754 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:27.757 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.756+0000 7f40e7fff640 1 -- 172.21.15.66:0/1956436670 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3830227941 0 0) 0x7f40ec004eb0 con 0x7f41001430c0 2024-03-25T18:06:27.757 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.756+0000 7f40e7fff640 1 -- 172.21.15.66:0/1956436670 >> v1:172.21.15.66:6790/0 conn(0x7f41000a98f0 legacy=0x7f4100130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:27.757 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.756+0000 7f40e7fff640 1 -- 172.21.15.66:0/1956436670 >> v1:172.21.15.66:6789/0 conn(0x7f410013f710 legacy=0x7f41001d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:27.757 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.756+0000 7f40e7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:06:27.757 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.756+0000 7f40e7fff640 1 monclient: found mon.b 2024-03-25T18:06:27.757 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.756+0000 7f40e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:27.757 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.756+0000 7f40e7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:06:27.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.756+0000 7f40e7fff640 1 -- 172.21.15.66:0/1956436670 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4100135d80 con 0x7f41001430c0 2024-03-25T18:06:27.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.756+0000 7f40e7fff640 10 monclient: _finish_auth 0 2024-03-25T18:06:27.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.756+0000 7f40e7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:06:27.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.756+0000 7f40e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:27.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.756+0000 7f4105a74640 5 monclient: authenticate success, global_id 4336 2024-03-25T18:06:27.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.756+0000 7f4105a74640 10 monclient: _renew_subs 2024-03-25T18:06:27.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.756+0000 7f4105a74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:06:27.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.756+0000 7f4105a74640 1 -- 172.21.15.66:0/1956436670 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4100131660 con 0x7f41001430c0 2024-03-25T18:06:27.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.757+0000 7f4105a74640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:06:27.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.757+0000 7f4105a74640 10 monclient: _renew_subs 2024-03-25T18:06:27.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.757+0000 7f4105a74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:06:27.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.757+0000 7f4105a74640 1 -- 172.21.15.66:0/1956436670 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f41000b7a80 con 0x7f41001430c0 2024-03-25T18:06:27.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.757+0000 7f4105a74640 1 librados: init done 2024-03-25T18:06:27.759 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.758+0000 7f4105a74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:27.759 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.758+0000 7f4105a74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:27.759 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.758+0000 7f4105a74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:06:27.759 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:27.758+0000 7f4105a74640 1 -- 172.21.15.66:0/1956436670 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4100131a00 con 0x7f41001430c0 2024-03-25T18:06:28.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.041+0000 7f40e7fff640 1 -- 172.21.15.66:0/1956436670 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40f80020a0 con 0x7f41001430c0 2024-03-25T18:06:28.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.041+0000 7f40e7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:28.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.041+0000 7f40e7fff640 1 -- 172.21.15.66:0/1956436670 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f40f8003990 con 0x7f41001430c0 2024-03-25T18:06:28.042 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.041+0000 7f40e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:28.042 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.041+0000 7f40e7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:28.042 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.041+0000 7f40e7fff640 10 monclient: dump: 2024-03-25T18:06:28.042 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:28.042 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:28.042 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:28.042 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:28.042 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:28.042 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:28.042 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:28.042 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:28.042 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:28.042 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:28.042 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.041+0000 7f40fdffb640 4 set_mon_vals no callback set 2024-03-25T18:06:28.043 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.042+0000 7f40e7fff640 1 -- 172.21.15.66:0/1956436670 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f40f8003bb0 con 0x7f41001430c0 2024-03-25T18:06:28.043 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.043+0000 7f40ff7fe640 10 monclient: get_auth_request con 0x7f40ec06e0c0 auth_method 0 2024-03-25T18:06:28.043 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.043+0000 7f40e7fff640 10 client.4336.objecter ms_handle_connect 0x7f40ec06e0c0 2024-03-25T18:06:28.048 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.047+0000 7f40e7fff640 1 -- 172.21.15.66:0/1956436670 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f40f80548b0 con 0x7f41001430c0 2024-03-25T18:06:28.048 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.047+0000 7f40e7fff640 10 client.4336.objecter ms_dispatch 0x7f4100000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:06:28.048 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.047+0000 7f40e7fff640 3 client.4336.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:06:28.048 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.047+0000 7f40e7fff640 3 client.4336.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:06:28.048 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.047+0000 7f40e7fff640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-25T18:06:28.049 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.049+0000 7f40e7fff640 1 -- 172.21.15.66:0/1956436670 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f40f80501f0 con 0x7f41001430c0 2024-03-25T18:06:28.049 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.049+0000 7f40e7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:28.049 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.049+0000 7f40e7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:06:28.306 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.306+0000 7f4105a74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:28.306 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.306+0000 7f4105a74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:28.306 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.306+0000 7f4105a74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:06:28.306 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.306+0000 7f4105a74640 1 -- 172.21.15.66:0/1956436670 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4100135200 con 0x7f41001430c0 2024-03-25T18:06:28.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.335+0000 7f40e7fff640 1 -- 172.21.15.66:0/1956436670 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f40f80522f0 con 0x7f41001430c0 2024-03-25T18:06:28.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.335+0000 7f40e7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:28.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.335+0000 7f40e7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:06:28.335 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:06:28.336 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:06:28.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.337+0000 7f4105a74640 10 librados: watch_flush enter 2024-03-25T18:06:28.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.337+0000 7f4105a74640 10 librados: watch_flush exit 2024-03-25T18:06:28.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.337+0000 7f4105a74640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-25T18:06:28.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.337+0000 7f4105a74640 10 client.4336.objecter successfully canceled tick 2024-03-25T18:06:28.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.337+0000 7f4105a74640 1 -- 172.21.15.66:0/1956436670 >> v1:172.21.15.193:6800/2275676803 conn(0x7f40ec06e0c0 legacy=0x7f40ec070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:28.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.337+0000 7f4105a74640 10 monclient: shutdown 2024-03-25T18:06:28.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.337+0000 7f4105a74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:28.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.337+0000 7f4105a74640 1 -- 172.21.15.66:0/1956436670 >> v1:172.21.15.193:6789/0 conn(0x7f41001430c0 legacy=0x7f41001e3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:28.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.337+0000 7f4105a74640 1 -- 172.21.15.66:0/1956436670 shutdown_connections 2024-03-25T18:06:28.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.337+0000 7f4105a74640 1 -- 172.21.15.66:0/1956436670 >> 172.21.15.66:0/1956436670 conn(0x7f41000b7ea0 msgr2=0x7f4100074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:28.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.337+0000 7f4105a74640 1 -- 172.21.15.66:0/1956436670 shutdown_connections 2024-03-25T18:06:28.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.337+0000 7f4105a74640 1 -- 172.21.15.66:0/1956436670 wait complete. 2024-03-25T18:06:28.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:28.337+0000 7f4105a74640 1 librados: shutdown 2024-03-25T18:06:28.350 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:06:35.157 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:06:35.278 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.276+0000 7fdba90f6640 10 monclient: get_monmap_and_config 2024-03-25T18:06:35.278 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.276+0000 7fdba90f6640 10 monclient: build_initial_monmap 2024-03-25T18:06:35.278 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.276+0000 7fdba90f6640 10 monclient: monmap: 2024-03-25T18:06:35.278 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:06:35.278 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:35.278 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:06:35.278353+0000 2024-03-25T18:06:35.278 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:06:35.278353+0000 2024-03-25T18:06:35.278 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:35.278 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:35.279 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:06:35.279 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:06:35.279 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:06:35.279 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:35.280 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.277+0000 7fdba90f6640 1 Processor -- start 2024-03-25T18:06:35.280 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.277+0000 7fdba90f6640 1 -- start start 2024-03-25T18:06:35.280 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.277+0000 7fdba90f6640 10 monclient: init 2024-03-25T18:06:35.280 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.278+0000 7fdba90f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:35.280 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.278+0000 7fdba90f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:06:35.280 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.278+0000 7fdba90f6640 10 monclient(hunting): picked mon.noname-a con 0x7fdba40a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:06:35.280 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.278+0000 7fdba90f6640 10 monclient(hunting): picked mon.noname-b con 0x7fdba413f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:06:35.280 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.278+0000 7fdba90f6640 10 monclient(hunting): picked mon.noname-c con 0x7fdba41430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:06:35.281 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.278+0000 7fdba90f6640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba4142520 con 0x7fdba40a98f0 2024-03-25T18:06:35.281 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.278+0000 7fdba90f6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba40b28a0 con 0x7fdba41430c0 2024-03-25T18:06:35.281 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.278+0000 7fdba90f6640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba40b2a30 con 0x7fdba413f710 2024-03-25T18:06:35.281 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.278+0000 7fdba90f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:35.281 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.278+0000 7fdba90f6640 10 monclient(hunting): authenticate will time out at 7144.338823s 2024-03-25T18:06:35.287 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.286+0000 7fdba259c640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7fdba413f710 0x7fdba41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:49234/0 (socket says 172.21.15.66:49234) 2024-03-25T18:06:35.287 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.286+0000 7fdba259c640 1 -- 172.21.15.66:0/3376472200 learned_addr learned my addr 172.21.15.66:0/3376472200 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:06:35.287 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.286+0000 7fdba2d9d640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7fdba40a98f0 0x7fdba40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:48934/0 (socket says 172.21.15.66:48934) 2024-03-25T18:06:35.299 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.297+0000 7fdba1d9b640 1 -- 172.21.15.66:0/3376472200 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3188011745 0 0) 0x7fdba40b2a30 con 0x7fdba413f710 2024-03-25T18:06:35.299 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.297+0000 7fdba1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:06:35.299 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.297+0000 7fdba1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:35.299 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.297+0000 7fdba1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:35.299 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.297+0000 7fdba1d9b640 10 monclient(hunting): my global_id is 4343 2024-03-25T18:06:35.299 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.297+0000 7fdba1d9b640 1 -- 172.21.15.66:0/3376472200 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb8c003520 con 0x7fdba413f710 2024-03-25T18:06:35.300 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.299+0000 7fdba1d9b640 1 -- 172.21.15.66:0/3376472200 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2098177443 0 0) 0x7fdba4142520 con 0x7fdba40a98f0 2024-03-25T18:06:35.300 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.299+0000 7fdba1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:06:35.301 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.299+0000 7fdba1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:35.301 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.299+0000 7fdba1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:35.301 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.299+0000 7fdba1d9b640 10 monclient(hunting): my global_id is 4344 2024-03-25T18:06:35.301 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.299+0000 7fdba1d9b640 1 -- 172.21.15.66:0/3376472200 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdba40b2a30 con 0x7fdba40a98f0 2024-03-25T18:06:35.313 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.311+0000 7fdba1d9b640 1 -- 172.21.15.66:0/3376472200 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1156347173 0 0) 0x7fdb8c003520 con 0x7fdba413f710 2024-03-25T18:06:35.313 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.311+0000 7fdba1d9b640 1 -- 172.21.15.66:0/3376472200 >> v1:172.21.15.193:6789/0 conn(0x7fdba41430c0 legacy=0x7fdba414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:35.313 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.311+0000 7fdba1d9b640 1 -- 172.21.15.66:0/3376472200 >> v1:172.21.15.66:6789/0 conn(0x7fdba40a98f0 legacy=0x7fdba40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:35.313 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.311+0000 7fdba1d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:06:35.313 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.311+0000 7fdba1d9b640 1 monclient: found mon.noname-b 2024-03-25T18:06:35.313 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.311+0000 7fdba1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:35.313 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.311+0000 7fdba1d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.66:6790/0 2024-03-25T18:06:35.313 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.311+0000 7fdba1d9b640 1 -- 172.21.15.66:0/3376472200 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba40b44c0 con 0x7fdba413f710 2024-03-25T18:06:35.313 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.311+0000 7fdba1d9b640 10 monclient: _finish_auth 0 2024-03-25T18:06:35.313 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.311+0000 7fdba1d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:06:35.314 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.311+0000 7fdba1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:35.314 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.311+0000 7fdba90f6640 5 monclient: authenticate success, global_id 4343 2024-03-25T18:06:35.314 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.311+0000 7fdba90f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:35.314 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.311+0000 7fdba1d9b640 1 -- 172.21.15.66:0/3376472200 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fdb98001690 con 0x7fdba413f710 2024-03-25T18:06:35.314 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.311+0000 7fdba1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:35.314 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.311+0000 7fdba1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:06:35.314 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.311+0000 7fdba1d9b640 10 monclient: dump: 2024-03-25T18:06:35.314 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:35.314 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:35.314 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:35.314 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:35.314 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:35.314 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:35.314 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:35.314 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:35.315 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:35.315 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:35.315 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.311+0000 7fdba90f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:35.317 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.316+0000 7fdba1d9b640 1 -- 172.21.15.66:0/3376472200 <== mon.2 v1:172.21.15.66:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb98001870 con 0x7fdba413f710 2024-03-25T18:06:35.317 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.316+0000 7fdba1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:35.318 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.316+0000 7fdba1d9b640 1 -- 172.21.15.66:0/3376472200 <== mon.2 v1:172.21.15.66:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fdb98001c80 con 0x7fdba413f710 2024-03-25T18:06:35.318 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.316+0000 7fdba1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:35.318 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.316+0000 7fdba1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:06:35.318 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.316+0000 7fdba1d9b640 10 monclient: dump: 2024-03-25T18:06:35.318 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:35.318 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:35.318 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:35.318 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:35.318 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:35.318 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:35.318 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:35.318 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:35.319 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:35.319 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:35.319 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.316+0000 7fdba90f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:06:35.319 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.316+0000 7fdba90f6640 4 set_mon_vals no callback set 2024-03-25T18:06:35.321 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.317+0000 7fdba90f6640 10 monclient: shutdown 2024-03-25T18:06:35.321 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.317+0000 7fdba90f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:35.321 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.317+0000 7fdba90f6640 1 -- 172.21.15.66:0/3376472200 >> v1:172.21.15.66:6790/0 conn(0x7fdba413f710 legacy=0x7fdba41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:35.321 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.317+0000 7fdba90f6640 1 -- 172.21.15.66:0/3376472200 shutdown_connections 2024-03-25T18:06:35.321 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.317+0000 7fdba90f6640 1 -- 172.21.15.66:0/3376472200 >> 172.21.15.66:0/3376472200 conn(0x7fdba40b7ea0 msgr2=0x7fdba40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:35.321 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.317+0000 7fdba90f6640 1 -- 172.21.15.66:0/3376472200 shutdown_connections 2024-03-25T18:06:35.321 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.317+0000 7fdba90f6640 1 -- 172.21.15.66:0/3376472200 wait complete. 2024-03-25T18:06:35.322 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.318+0000 7fdba90f6640 10 monclient: build_initial_monmap 2024-03-25T18:06:35.322 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.318+0000 7fdba90f6640 10 monclient: monmap: 2024-03-25T18:06:35.322 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:06:35.322 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:06:35.322 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:06:35.322 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:06:35.322 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:35.322 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:06:35.322 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:06:35.322 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:06:35.322 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:06:35.322 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:35.322 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.318+0000 7fdba90f6640 1 librados: starting msgr at 2024-03-25T18:06:35.322 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.318+0000 7fdba90f6640 1 librados: starting objecter 2024-03-25T18:06:35.322 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.319+0000 7fdba90f6640 1 Processor -- start 2024-03-25T18:06:35.323 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.319+0000 7fdba90f6640 1 -- start start 2024-03-25T18:06:35.323 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.319+0000 7fdba90f6640 1 librados: setting wanted keys 2024-03-25T18:06:35.323 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.319+0000 7fdba90f6640 1 librados: calling monclient init 2024-03-25T18:06:35.323 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.319+0000 7fdba90f6640 10 monclient: init 2024-03-25T18:06:35.323 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.319+0000 7fdba90f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:35.323 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.319+0000 7fdba90f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:06:35.323 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.319+0000 7fdba90f6640 10 monclient(hunting): picked mon.noname-c con 0x7fdba40a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:06:35.323 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.319+0000 7fdba90f6640 10 monclient(hunting): picked mon.noname-a con 0x7fdba413f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:06:35.323 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.319+0000 7fdba90f6640 10 monclient(hunting): picked mon.noname-b con 0x7fdba41430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:06:35.323 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.319+0000 7fdba90f6640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba4138cb0 con 0x7fdba413f710 2024-03-25T18:06:35.323 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.319+0000 7fdba90f6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba40ab920 con 0x7fdba41430c0 2024-03-25T18:06:35.323 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.319+0000 7fdba90f6640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba4138010 con 0x7fdba40a98f0 2024-03-25T18:06:35.323 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.319+0000 7fdba90f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:35.323 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.319+0000 7fdba90f6640 10 monclient(hunting): authenticate will time out at 7144.380053s 2024-03-25T18:06:35.329 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.328+0000 7fdba359e640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fdba41430c0 0x7fdba4134010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:40478/0 (socket says 172.21.15.66:40478) 2024-03-25T18:06:35.329 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.328+0000 7fdba359e640 1 -- 172.21.15.66:0/1162113750 learned_addr learned my addr 172.21.15.66:0/1162113750 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:06:35.330 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.329+0000 7fdb8b7fe640 10 client.?.objecter ms_handle_connect 0x7fdba41430c0 2024-03-25T18:06:35.331 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.329+0000 7fdb8b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:35.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.337+0000 7fdb8b7fe640 1 -- 172.21.15.66:0/1162113750 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2582372706 0 0) 0x7fdba40ab920 con 0x7fdba41430c0 2024-03-25T18:06:35.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.337+0000 7fdb8b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:06:35.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.337+0000 7fdb8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:35.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.337+0000 7fdb8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:35.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.337+0000 7fdb8b7fe640 10 monclient(hunting): my global_id is 4342 2024-03-25T18:06:35.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.337+0000 7fdb8b7fe640 1 -- 172.21.15.66:0/1162113750 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb90004eb0 con 0x7fdba41430c0 2024-03-25T18:06:35.343 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.341+0000 7fdb8b7fe640 10 client.?.objecter ms_handle_connect 0x7fdba413f710 2024-03-25T18:06:35.343 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.341+0000 7fdb8b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:35.344 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.343+0000 7fdb8b7fe640 10 client.?.objecter ms_handle_connect 0x7fdba40a98f0 2024-03-25T18:06:35.344 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.343+0000 7fdb8b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:35.346 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.344+0000 7fdb8b7fe640 1 -- 172.21.15.66:0/1162113750 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3415344013 0 0) 0x7fdba4138cb0 con 0x7fdba413f710 2024-03-25T18:06:35.346 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.344+0000 7fdb8b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:06:35.346 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.344+0000 7fdb8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:35.346 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.344+0000 7fdb8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:35.346 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.344+0000 7fdb8b7fe640 10 monclient(hunting): my global_id is 4347 2024-03-25T18:06:35.346 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.344+0000 7fdb8b7fe640 1 -- 172.21.15.66:0/1162113750 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdba40ab920 con 0x7fdba413f710 2024-03-25T18:06:35.359 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.358+0000 7fdb8b7fe640 1 -- 172.21.15.66:0/1162113750 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3287901247 0 0) 0x7fdb90004eb0 con 0x7fdba41430c0 2024-03-25T18:06:35.359 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.358+0000 7fdb8b7fe640 1 -- 172.21.15.66:0/1162113750 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdba4138cb0 con 0x7fdba41430c0 2024-03-25T18:06:35.359 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.358+0000 7fdb8b7fe640 1 -- 172.21.15.66:0/1162113750 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fdb9c001c90 con 0x7fdba41430c0 2024-03-25T18:06:35.359 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.358+0000 7fdb8b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:35.359 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.358+0000 7fdb8b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:06:35.359 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.358+0000 7fdb8b7fe640 10 monclient(hunting): dump: 2024-03-25T18:06:35.359 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:35.359 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:35.360 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:35.360 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:35.360 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:35.360 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:35.360 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:35.360 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:35.360 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:35.360 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:35.360 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.358+0000 7fdb8b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:06:35.361 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.359+0000 7fdb8b7fe640 1 -- 172.21.15.66:0/1162113750 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2130970560 0 0) 0x7fdba40ab920 con 0x7fdba413f710 2024-03-25T18:06:35.361 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.359+0000 7fdb8b7fe640 1 -- 172.21.15.66:0/1162113750 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb90004eb0 con 0x7fdba413f710 2024-03-25T18:06:35.361 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.359+0000 7fdb8b7fe640 1 -- 172.21.15.66:0/1162113750 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fdb98001b80 con 0x7fdba413f710 2024-03-25T18:06:35.361 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.359+0000 7fdb8b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:35.361 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.359+0000 7fdb8b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:06:35.361 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.359+0000 7fdb8b7fe640 10 monclient(hunting): dump: 2024-03-25T18:06:35.361 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:35.361 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:35.361 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:35.361 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:35.361 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:35.361 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:35.361 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:35.362 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:35.362 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:35.362 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:35.363 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.362+0000 7fdb8b7fe640 1 -- 172.21.15.66:0/1162113750 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3577820491 0 0) 0x7fdba4138010 con 0x7fdba40a98f0 2024-03-25T18:06:35.363 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.362+0000 7fdb8b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:06:35.363 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.362+0000 7fdb8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:35.363 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.362+0000 7fdb8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:35.363 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.362+0000 7fdb8b7fe640 10 monclient(hunting): my global_id is 4346 2024-03-25T18:06:35.363 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.362+0000 7fdb8b7fe640 1 -- 172.21.15.66:0/1162113750 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdba40ab920 con 0x7fdba40a98f0 2024-03-25T18:06:35.368 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.366+0000 7fdb8b7fe640 1 -- 172.21.15.66:0/1162113750 <== mon.0 v1:172.21.15.66:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3879202660 0 0) 0x7fdb90004eb0 con 0x7fdba413f710 2024-03-25T18:06:35.368 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.366+0000 7fdb8b7fe640 1 -- 172.21.15.66:0/1162113750 >> v1:172.21.15.66:6790/0 conn(0x7fdba40a98f0 legacy=0x7fdba41d7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:35.368 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.366+0000 7fdb8b7fe640 1 -- 172.21.15.66:0/1162113750 >> v1:172.21.15.193:6789/0 conn(0x7fdba41430c0 legacy=0x7fdba4134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:35.368 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.366+0000 7fdb8b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:06:35.368 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.366+0000 7fdb8b7fe640 1 monclient: found mon.a 2024-03-25T18:06:35.368 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.366+0000 7fdb8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:35.368 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.366+0000 7fdb8b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:06:35.368 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.366+0000 7fdb8b7fe640 1 -- 172.21.15.66:0/1162113750 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba4137d50 con 0x7fdba413f710 2024-03-25T18:06:35.368 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.366+0000 7fdb8b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:06:35.368 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.366+0000 7fdb8b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:06:35.368 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.366+0000 7fdb8b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:35.369 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.366+0000 7fdba90f6640 5 monclient: authenticate success, global_id 4347 2024-03-25T18:06:35.369 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.366+0000 7fdba90f6640 10 monclient: _renew_subs 2024-03-25T18:06:35.369 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.366+0000 7fdba90f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:06:35.369 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.366+0000 7fdba90f6640 1 -- 172.21.15.66:0/1162113750 --> v1:172.21.15.66:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdba4137890 con 0x7fdba413f710 2024-03-25T18:06:35.369 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.366+0000 7fdba90f6640 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:06:35.369 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.366+0000 7fdba90f6640 10 monclient: _renew_subs 2024-03-25T18:06:35.369 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.366+0000 7fdba90f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:06:35.369 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.366+0000 7fdba90f6640 1 -- 172.21.15.66:0/1162113750 --> v1:172.21.15.66:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdba4134900 con 0x7fdba413f710 2024-03-25T18:06:35.369 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.366+0000 7fdba90f6640 1 librados: init done 2024-03-25T18:06:35.370 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.367+0000 7fdba90f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:35.370 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.368+0000 7fdba90f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:35.370 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.368+0000 7fdba90f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:06:35.370 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.368+0000 7fdba90f6640 1 -- 172.21.15.66:0/1162113750 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdba40627b0 con 0x7fdba413f710 2024-03-25T18:06:35.682 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.681+0000 7fdb8b7fe640 1 -- 172.21.15.66:0/1162113750 <== mon.0 v1:172.21.15.66:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb98001d20 con 0x7fdba413f710 2024-03-25T18:06:35.682 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.681+0000 7fdb8b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:35.682 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.681+0000 7fdb8b7fe640 1 -- 172.21.15.66:0/1162113750 <== mon.0 v1:172.21.15.66:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fdb98004d90 con 0x7fdba413f710 2024-03-25T18:06:35.682 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.681+0000 7fdb8b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:35.682 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.681+0000 7fdb8b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:06:35.683 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.681+0000 7fdb8b7fe640 10 monclient: dump: 2024-03-25T18:06:35.683 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:35.683 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:35.683 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:35.683 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:35.683 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:35.683 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:35.683 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:35.683 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:35.683 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:35.683 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:35.684 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.681+0000 7fdba159a640 4 set_mon_vals no callback set 2024-03-25T18:06:35.684 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.681+0000 7fdb8b7fe640 1 -- 172.21.15.66:0/1162113750 <== mon.0 v1:172.21.15.66:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7fdb9801ad60 con 0x7fdba413f710 2024-03-25T18:06:35.684 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.682+0000 7fdba2d9d640 10 monclient: get_auth_request con 0x7fdb9006df00 auth_method 0 2024-03-25T18:06:35.684 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.682+0000 7fdb8b7fe640 10 client.4347.objecter ms_handle_connect 0x7fdb9006df00 2024-03-25T18:06:35.692 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.691+0000 7fdb8b7fe640 1 -- 172.21.15.66:0/1162113750 <== mon.0 v1:172.21.15.66:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7fdb98054920 con 0x7fdba413f710 2024-03-25T18:06:35.692 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.691+0000 7fdb8b7fe640 10 client.4347.objecter ms_dispatch 0x7fdba4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:06:35.693 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.691+0000 7fdb8b7fe640 3 client.4347.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:06:35.693 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.691+0000 7fdb8b7fe640 3 client.4347.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:06:35.693 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.691+0000 7fdb8b7fe640 20 client.4347.objecter dump_active .. 0 homeless 2024-03-25T18:06:35.693 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.691+0000 7fdb8b7fe640 1 -- 172.21.15.66:0/1162113750 <== mon.0 v1:172.21.15.66:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdb98050430 con 0x7fdba413f710 2024-03-25T18:06:35.693 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.691+0000 7fdb8b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:35.693 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.691+0000 7fdb8b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:06:35.914 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.912+0000 7fdba90f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:35.914 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.912+0000 7fdba90f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:35.914 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.912+0000 7fdba90f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:06:35.914 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.912+0000 7fdba90f6640 1 -- 172.21.15.66:0/1162113750 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdba4069f40 con 0x7fdba413f710 2024-03-25T18:06:35.933 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.931+0000 7fdb8b7fe640 1 -- 172.21.15.66:0/1162113750 <== mon.0 v1:172.21.15.66:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7fdb98052360 con 0x7fdba413f710 2024-03-25T18:06:35.933 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.931+0000 7fdb8b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:35.933 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.931+0000 7fdb8b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:06:35.933 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:06:35.933 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:06:35.935 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.933+0000 7fdba90f6640 10 librados: watch_flush enter 2024-03-25T18:06:35.936 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.933+0000 7fdba90f6640 10 librados: watch_flush exit 2024-03-25T18:06:35.936 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.933+0000 7fdba90f6640 20 client.4347.objecter shutdown clearing up homeless session... 2024-03-25T18:06:35.936 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.933+0000 7fdba90f6640 10 client.4347.objecter successfully canceled tick 2024-03-25T18:06:35.936 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.933+0000 7fdba90f6640 1 -- 172.21.15.66:0/1162113750 >> v1:172.21.15.193:6800/2275676803 conn(0x7fdb9006df00 legacy=0x7fdb900703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:35.936 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.933+0000 7fdba90f6640 10 monclient: shutdown 2024-03-25T18:06:35.936 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.933+0000 7fdba90f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:35.936 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.933+0000 7fdba90f6640 1 -- 172.21.15.66:0/1162113750 >> v1:172.21.15.66:6789/0 conn(0x7fdba413f710 legacy=0x7fdba41385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:35.936 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.934+0000 7fdba90f6640 1 -- 172.21.15.66:0/1162113750 shutdown_connections 2024-03-25T18:06:35.936 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.934+0000 7fdba90f6640 1 -- 172.21.15.66:0/1162113750 >> 172.21.15.66:0/1162113750 conn(0x7fdba40b7ea0 msgr2=0x7fdba4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:35.936 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.934+0000 7fdba90f6640 1 -- 172.21.15.66:0/1162113750 shutdown_connections 2024-03-25T18:06:35.936 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.934+0000 7fdba90f6640 1 -- 172.21.15.66:0/1162113750 wait complete. 2024-03-25T18:06:35.936 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:35.934+0000 7fdba90f6640 1 librados: shutdown 2024-03-25T18:06:35.946 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:06:42.752 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:06:42.871 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.870+0000 7f85786f6640 10 monclient: get_monmap_and_config 2024-03-25T18:06:42.871 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.870+0000 7f85786f6640 10 monclient: build_initial_monmap 2024-03-25T18:06:42.871 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.870+0000 7f85786f6640 10 monclient: monmap: 2024-03-25T18:06:42.872 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:06:42.872 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:42.872 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:06:42.871807+0000 2024-03-25T18:06:42.872 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:06:42.871807+0000 2024-03-25T18:06:42.872 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:42.872 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:42.872 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:06:42.872 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:06:42.872 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:06:42.872 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:42.873 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.870+0000 7f85786f6640 1 Processor -- start 2024-03-25T18:06:42.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.871+0000 7f85786f6640 1 -- start start 2024-03-25T18:06:42.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.871+0000 7f85786f6640 10 monclient: init 2024-03-25T18:06:42.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.871+0000 7f85786f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:42.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.871+0000 7f85786f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:06:42.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.871+0000 7f85786f6640 10 monclient(hunting): picked mon.noname-c con 0x7f85700a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:06:42.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.871+0000 7f85786f6640 10 monclient(hunting): picked mon.noname-a con 0x7f857013f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:06:42.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.871+0000 7f85786f6640 10 monclient(hunting): picked mon.noname-b con 0x7f85701430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:06:42.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.871+0000 7f85786f6640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8570142520 con 0x7f857013f710 2024-03-25T18:06:42.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.871+0000 7f85786f6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85700b28a0 con 0x7f85700a98f0 2024-03-25T18:06:42.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.871+0000 7f85786f6640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85700b2a30 con 0x7f85701430c0 2024-03-25T18:06:42.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.871+0000 7f85786f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:42.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.871+0000 7f85786f6640 10 monclient(hunting): authenticate will time out at 7151.931956s 2024-03-25T18:06:42.878 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.876+0000 7f8576492640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f85700a98f0 0x7f85700a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:45040/0 (socket says 172.21.15.66:45040) 2024-03-25T18:06:42.878 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.877+0000 7f8576492640 1 -- 172.21.15.66:0/3731005276 learned_addr learned my addr 172.21.15.66:0/3731005276 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:06:42.896 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.895+0000 7f8575490640 1 -- 172.21.15.66:0/3731005276 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 763399353 0 0) 0x7f85700b28a0 con 0x7f85700a98f0 2024-03-25T18:06:42.897 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.895+0000 7f8575490640 10 monclient(hunting): _negotiate 2024-03-25T18:06:42.897 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.895+0000 7f8575490640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:42.897 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.895+0000 7f8575490640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:42.897 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.895+0000 7f8575490640 10 monclient(hunting): my global_id is 4345 2024-03-25T18:06:42.897 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.895+0000 7f8575490640 1 -- 172.21.15.66:0/3731005276 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f855c003520 con 0x7f85700a98f0 2024-03-25T18:06:42.906 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.905+0000 7f8575490640 1 -- 172.21.15.66:0/3731005276 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1482890109 0 0) 0x7f8570142520 con 0x7f857013f710 2024-03-25T18:06:42.906 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.905+0000 7f8575490640 10 monclient(hunting): _negotiate 2024-03-25T18:06:42.906 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.905+0000 7f8575490640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:42.906 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.905+0000 7f8575490640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:42.906 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.905+0000 7f8575490640 10 monclient(hunting): my global_id is 4350 2024-03-25T18:06:42.906 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.905+0000 7f8575490640 1 -- 172.21.15.66:0/3731005276 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85700b28a0 con 0x7f857013f710 2024-03-25T18:06:42.907 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.905+0000 7f8575490640 1 -- 172.21.15.66:0/3731005276 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 628579447 0 0) 0x7f85700b2a30 con 0x7f85701430c0 2024-03-25T18:06:42.907 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.905+0000 7f8575490640 10 monclient(hunting): _negotiate 2024-03-25T18:06:42.907 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.905+0000 7f8575490640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:42.907 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.905+0000 7f8575490640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:42.907 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.905+0000 7f8575490640 10 monclient(hunting): my global_id is 4349 2024-03-25T18:06:42.907 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.905+0000 7f8575490640 1 -- 172.21.15.66:0/3731005276 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8570142520 con 0x7f85701430c0 2024-03-25T18:06:42.908 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.906+0000 7f8575490640 1 -- 172.21.15.66:0/3731005276 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3011693530 0 0) 0x7f855c003520 con 0x7f85700a98f0 2024-03-25T18:06:42.908 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.906+0000 7f8575490640 1 -- 172.21.15.66:0/3731005276 >> v1:172.21.15.66:6790/0 conn(0x7f85701430c0 legacy=0x7f857014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:42.908 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.906+0000 7f8575490640 1 -- 172.21.15.66:0/3731005276 >> v1:172.21.15.66:6789/0 conn(0x7f857013f710 legacy=0x7f85701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:42.908 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.906+0000 7f8575490640 10 monclient: _finish_hunting 0 2024-03-25T18:06:42.908 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.906+0000 7f8575490640 1 monclient: found mon.noname-c 2024-03-25T18:06:42.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.906+0000 7f8575490640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:42.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.906+0000 7f8575490640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:06:42.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.906+0000 7f8575490640 1 -- 172.21.15.66:0/3731005276 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85700b44c0 con 0x7f85700a98f0 2024-03-25T18:06:42.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.906+0000 7f8575490640 10 monclient: _finish_auth 0 2024-03-25T18:06:42.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.906+0000 7f8575490640 10 monclient: _check_auth_tickets 2024-03-25T18:06:42.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.906+0000 7f8575490640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:42.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.906+0000 7f8575490640 1 -- 172.21.15.66:0/3731005276 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f856c001690 con 0x7f85700a98f0 2024-03-25T18:06:42.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.906+0000 7f8575490640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:42.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.906+0000 7f8575490640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:06:42.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.906+0000 7f8575490640 10 monclient: dump: 2024-03-25T18:06:42.909 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:42.909 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:42.909 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:42.909 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:42.910 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:42.910 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:42.910 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:42.910 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:42.910 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:42.910 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:42.910 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.906+0000 7f85786f6640 5 monclient: authenticate success, global_id 4345 2024-03-25T18:06:42.910 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.906+0000 7f85786f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:42.913 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.912+0000 7f8575490640 1 -- 172.21.15.66:0/3731005276 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f856c0017f0 con 0x7f85700a98f0 2024-03-25T18:06:42.914 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.912+0000 7f8575490640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:42.914 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.912+0000 7f8575490640 1 -- 172.21.15.66:0/3731005276 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f856c001c00 con 0x7f85700a98f0 2024-03-25T18:06:42.914 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.912+0000 7f85786f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:06:42.914 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.912+0000 7f85786f6640 4 set_mon_vals no callback set 2024-03-25T18:06:42.914 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.912+0000 7f8575490640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:42.914 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.912+0000 7f8575490640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:42.914 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.912+0000 7f8575490640 10 monclient: dump: 2024-03-25T18:06:42.914 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:42.914 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:42.914 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:42.914 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:42.915 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:42.915 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:42.915 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:42.915 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:42.915 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:42.915 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:42.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.913+0000 7f85786f6640 10 monclient: shutdown 2024-03-25T18:06:42.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.913+0000 7f85786f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:42.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.913+0000 7f85786f6640 1 -- 172.21.15.66:0/3731005276 >> v1:172.21.15.193:6789/0 conn(0x7f85700a98f0 legacy=0x7f85700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:42.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.913+0000 7f85786f6640 1 -- 172.21.15.66:0/3731005276 shutdown_connections 2024-03-25T18:06:42.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.913+0000 7f85786f6640 1 -- 172.21.15.66:0/3731005276 >> 172.21.15.66:0/3731005276 conn(0x7f85700b7ea0 msgr2=0x7f85700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:42.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.913+0000 7f85786f6640 1 -- 172.21.15.66:0/3731005276 shutdown_connections 2024-03-25T18:06:42.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.914+0000 7f85786f6640 1 -- 172.21.15.66:0/3731005276 wait complete. 2024-03-25T18:06:42.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.914+0000 7f85786f6640 10 monclient: build_initial_monmap 2024-03-25T18:06:42.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.915+0000 7f85786f6640 10 monclient: monmap: 2024-03-25T18:06:42.919 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:06:42.919 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:06:42.919 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:06:42.919 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:06:42.919 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:42.919 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:06:42.919 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:06:42.919 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:06:42.919 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:06:42.919 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:42.919 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.915+0000 7f85786f6640 1 librados: starting msgr at 2024-03-25T18:06:42.919 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.915+0000 7f85786f6640 1 librados: starting objecter 2024-03-25T18:06:42.920 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.915+0000 7f85786f6640 1 Processor -- start 2024-03-25T18:06:42.920 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.915+0000 7f85786f6640 1 -- start start 2024-03-25T18:06:42.920 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.915+0000 7f85786f6640 1 librados: setting wanted keys 2024-03-25T18:06:42.920 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.915+0000 7f85786f6640 1 librados: calling monclient init 2024-03-25T18:06:42.920 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.915+0000 7f85786f6640 10 monclient: init 2024-03-25T18:06:42.920 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.916+0000 7f85786f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:42.920 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.916+0000 7f85786f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:06:42.920 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.916+0000 7f85786f6640 10 monclient(hunting): picked mon.noname-b con 0x7f85700a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:06:42.920 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.916+0000 7f85786f6640 10 monclient(hunting): picked mon.noname-c con 0x7f857013f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:06:42.920 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.916+0000 7f85786f6640 10 monclient(hunting): picked mon.noname-a con 0x7f85701430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:06:42.920 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.916+0000 7f85786f6640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8570138a50 con 0x7f85701430c0 2024-03-25T18:06:42.920 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.916+0000 7f85786f6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85700ab9e0 con 0x7f85700a98f0 2024-03-25T18:06:42.921 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.916+0000 7f85786f6640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85701d7880 con 0x7f857013f710 2024-03-25T18:06:42.921 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.916+0000 7f85786f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:42.921 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.916+0000 7f85786f6640 10 monclient(hunting): authenticate will time out at 7151.976458s 2024-03-25T18:06:42.921 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.920+0000 7f8576492640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f85700a98f0 0x7f8570138340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:45076/0 (socket says 172.21.15.66:45076) 2024-03-25T18:06:42.921 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.920+0000 7f8576492640 1 -- 172.21.15.66:0/369664029 learned_addr learned my addr 172.21.15.66:0/369664029 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:06:42.926 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.925+0000 7f8566ffd640 10 client.?.objecter ms_handle_connect 0x7f85700a98f0 2024-03-25T18:06:42.926 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.925+0000 7f8566ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:42.933 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.931+0000 7f8566ffd640 1 -- 172.21.15.66:0/369664029 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4186031507 0 0) 0x7f85700ab9e0 con 0x7f85700a98f0 2024-03-25T18:06:42.933 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.931+0000 7f8566ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:06:42.933 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.931+0000 7f8566ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:42.933 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.931+0000 7f8566ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:42.933 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.931+0000 7f8566ffd640 10 monclient(hunting): my global_id is 4348 2024-03-25T18:06:42.933 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.931+0000 7f8566ffd640 1 -- 172.21.15.66:0/369664029 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8554004eb0 con 0x7f85700a98f0 2024-03-25T18:06:42.934 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.933+0000 7f8566ffd640 10 client.?.objecter ms_handle_connect 0x7f85701430c0 2024-03-25T18:06:42.934 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.933+0000 7f8566ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:42.944 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.943+0000 7f8566ffd640 10 client.?.objecter ms_handle_connect 0x7f857013f710 2024-03-25T18:06:42.944 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.943+0000 7f8566ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:42.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.945+0000 7f8566ffd640 1 -- 172.21.15.66:0/369664029 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 79796415 0 0) 0x7f8570138a50 con 0x7f85701430c0 2024-03-25T18:06:42.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.945+0000 7f8566ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:06:42.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.945+0000 7f8566ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:42.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.945+0000 7f8566ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:42.947 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.945+0000 7f8566ffd640 10 monclient(hunting): my global_id is 4353 2024-03-25T18:06:42.947 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.945+0000 7f8566ffd640 1 -- 172.21.15.66:0/369664029 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85700ab9e0 con 0x7f85701430c0 2024-03-25T18:06:42.951 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.949+0000 7f8566ffd640 1 -- 172.21.15.66:0/369664029 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3155267852 0 0) 0x7f8554004eb0 con 0x7f85700a98f0 2024-03-25T18:06:42.951 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.949+0000 7f8566ffd640 1 -- 172.21.15.66:0/369664029 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8570138a50 con 0x7f85700a98f0 2024-03-25T18:06:42.951 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.949+0000 7f8566ffd640 1 -- 172.21.15.66:0/369664029 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f856c001bc0 con 0x7f85700a98f0 2024-03-25T18:06:42.951 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.949+0000 7f8566ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:42.951 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.949+0000 7f8566ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:06:42.951 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.949+0000 7f8566ffd640 10 monclient(hunting): dump: 2024-03-25T18:06:42.951 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:42.951 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:42.951 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:42.952 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:42.952 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:42.952 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:42.952 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:42.952 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:42.952 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:42.952 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:42.952 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.949+0000 7f8566ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:06:42.959 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.958+0000 7f8566ffd640 1 -- 172.21.15.66:0/369664029 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559344974 0 0) 0x7f85701d7880 con 0x7f857013f710 2024-03-25T18:06:42.960 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.958+0000 7f8566ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:06:42.960 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.958+0000 7f8566ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:42.960 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.958+0000 7f8566ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:42.960 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.958+0000 7f8566ffd640 10 monclient(hunting): my global_id is 4352 2024-03-25T18:06:42.960 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.958+0000 7f8566ffd640 1 -- 172.21.15.66:0/369664029 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8554004eb0 con 0x7f857013f710 2024-03-25T18:06:42.966 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.964+0000 7f8566ffd640 1 -- 172.21.15.66:0/369664029 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3628493601 0 0) 0x7f8570138a50 con 0x7f85700a98f0 2024-03-25T18:06:42.966 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.964+0000 7f8566ffd640 1 -- 172.21.15.66:0/369664029 >> v1:172.21.15.66:6790/0 conn(0x7f857013f710 legacy=0x7f85701d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:42.966 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.964+0000 7f8566ffd640 1 -- 172.21.15.66:0/369664029 >> v1:172.21.15.66:6789/0 conn(0x7f85701430c0 legacy=0x7f85701e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:42.966 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.964+0000 7f8566ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:06:42.966 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.964+0000 7f8566ffd640 1 monclient: found mon.b 2024-03-25T18:06:42.966 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.964+0000 7f8566ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:42.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.964+0000 7f8566ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:06:42.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.964+0000 7f8566ffd640 1 -- 172.21.15.66:0/369664029 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85701d84b0 con 0x7f85700a98f0 2024-03-25T18:06:42.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.964+0000 7f8566ffd640 10 monclient: _finish_auth 0 2024-03-25T18:06:42.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.964+0000 7f8566ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:06:42.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.964+0000 7f8566ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:42.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.964+0000 7f85786f6640 5 monclient: authenticate success, global_id 4348 2024-03-25T18:06:42.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.964+0000 7f85786f6640 10 monclient: _renew_subs 2024-03-25T18:06:42.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.964+0000 7f85786f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:06:42.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.964+0000 7f85786f6640 1 -- 172.21.15.66:0/369664029 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85701d51f0 con 0x7f85700a98f0 2024-03-25T18:06:42.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.964+0000 7f85786f6640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:06:42.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.964+0000 7f85786f6640 10 monclient: _renew_subs 2024-03-25T18:06:42.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.964+0000 7f85786f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:06:42.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.964+0000 7f85786f6640 1 -- 172.21.15.66:0/369664029 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8570138e60 con 0x7f85700a98f0 2024-03-25T18:06:42.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.964+0000 7f85786f6640 1 librados: init done 2024-03-25T18:06:42.968 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.965+0000 7f85786f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:42.968 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.965+0000 7f85786f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:42.968 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.965+0000 7f85786f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:06:42.968 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:42.965+0000 7f85786f6640 1 -- 172.21.15.66:0/369664029 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85701d76e0 con 0x7f85700a98f0 2024-03-25T18:06:43.249 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.247+0000 7f8566ffd640 1 -- 172.21.15.66:0/369664029 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f856c0011f0 con 0x7f85700a98f0 2024-03-25T18:06:43.249 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.247+0000 7f8566ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:43.249 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.247+0000 7f8566ffd640 1 -- 172.21.15.66:0/369664029 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f856c004a90 con 0x7f85700a98f0 2024-03-25T18:06:43.249 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.247+0000 7f8566ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:43.249 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.247+0000 7f8566ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:43.250 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.247+0000 7f8566ffd640 10 monclient: dump: 2024-03-25T18:06:43.250 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:43.250 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:43.250 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:43.250 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:43.250 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:43.250 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:43.250 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:43.250 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:43.250 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:43.250 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:43.251 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.247+0000 7f8574c8f640 4 set_mon_vals no callback set 2024-03-25T18:06:43.251 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.248+0000 7f8566ffd640 1 -- 172.21.15.66:0/369664029 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f856c01aae0 con 0x7f85700a98f0 2024-03-25T18:06:43.251 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.249+0000 7f8575c91640 10 monclient: get_auth_request con 0x7f855406ddb0 auth_method 0 2024-03-25T18:06:43.252 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.249+0000 7f8566ffd640 10 client.4348.objecter ms_handle_connect 0x7f855406ddb0 2024-03-25T18:06:43.256 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.255+0000 7f8566ffd640 1 -- 172.21.15.66:0/369664029 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f856c054790 con 0x7f85700a98f0 2024-03-25T18:06:43.257 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.255+0000 7f8566ffd640 10 client.4348.objecter ms_dispatch 0x7f8570000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:06:43.257 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.255+0000 7f8566ffd640 3 client.4348.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:06:43.257 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.255+0000 7f8566ffd640 3 client.4348.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:06:43.257 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.255+0000 7f8566ffd640 20 client.4348.objecter dump_active .. 0 homeless 2024-03-25T18:06:43.257 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.255+0000 7f8566ffd640 1 -- 172.21.15.66:0/369664029 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f856c0500d0 con 0x7f85700a98f0 2024-03-25T18:06:43.257 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.255+0000 7f8566ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:43.257 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.255+0000 7f8566ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:06:43.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.519+0000 7f85786f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:43.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.519+0000 7f85786f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:43.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.519+0000 7f85786f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:06:43.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.519+0000 7f85786f6640 1 -- 172.21.15.66:0/369664029 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85701d7a10 con 0x7f85700a98f0 2024-03-25T18:06:43.607 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.606+0000 7f8566ffd640 1 -- 172.21.15.66:0/369664029 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f856c0521d0 con 0x7f85700a98f0 2024-03-25T18:06:43.608 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.606+0000 7f8566ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:43.608 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.606+0000 7f8566ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:06:43.608 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:06:43.608 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:06:43.611 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.609+0000 7f85786f6640 10 librados: watch_flush enter 2024-03-25T18:06:43.611 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.609+0000 7f85786f6640 10 librados: watch_flush exit 2024-03-25T18:06:43.611 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.609+0000 7f85786f6640 20 client.4348.objecter shutdown clearing up homeless session... 2024-03-25T18:06:43.611 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.609+0000 7f85786f6640 10 client.4348.objecter successfully canceled tick 2024-03-25T18:06:43.611 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.609+0000 7f85786f6640 1 -- 172.21.15.66:0/369664029 >> v1:172.21.15.193:6800/2275676803 conn(0x7f855406ddb0 legacy=0x7f8554070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:43.611 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.609+0000 7f85786f6640 10 monclient: shutdown 2024-03-25T18:06:43.611 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.609+0000 7f85786f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:43.611 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.609+0000 7f85786f6640 1 -- 172.21.15.66:0/369664029 >> v1:172.21.15.193:6789/0 conn(0x7f85700a98f0 legacy=0x7f8570138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:43.611 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.609+0000 7f85786f6640 1 -- 172.21.15.66:0/369664029 shutdown_connections 2024-03-25T18:06:43.611 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.609+0000 7f85786f6640 1 -- 172.21.15.66:0/369664029 >> 172.21.15.66:0/369664029 conn(0x7f85700b7ea0 msgr2=0x7f8570074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:43.612 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.609+0000 7f85786f6640 1 -- 172.21.15.66:0/369664029 shutdown_connections 2024-03-25T18:06:43.612 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.609+0000 7f85786f6640 1 -- 172.21.15.66:0/369664029 wait complete. 2024-03-25T18:06:43.612 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:43.609+0000 7f85786f6640 1 librados: shutdown 2024-03-25T18:06:43.621 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:06:43.970 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:03:14.951 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:44.025 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:03:15.005 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:44.056 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:03:15.037 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:44.079 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:03:15.038 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:44.088 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:03:15.068 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:44.096 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:03:15.054 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:44.099 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:03:15.080 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:44.124 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:03:15.082 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:44.128 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:03:15.109 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:06:50.428 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:06:50.537 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.536+0000 7f7bceac9640 10 monclient: get_monmap_and_config 2024-03-25T18:06:50.537 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.536+0000 7f7bceac9640 10 monclient: build_initial_monmap 2024-03-25T18:06:50.537 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.536+0000 7f7bceac9640 10 monclient: monmap: 2024-03-25T18:06:50.537 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:06:50.537 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:50.537 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:06:50.537638+0000 2024-03-25T18:06:50.537 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:06:50.537638+0000 2024-03-25T18:06:50.537 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:50.537 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:50.537 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:06:50.538 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:06:50.538 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:06:50.538 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:50.538 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.537+0000 7f7bceac9640 1 Processor -- start 2024-03-25T18:06:50.538 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.537+0000 7f7bceac9640 1 -- start start 2024-03-25T18:06:50.538 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.537+0000 7f7bceac9640 10 monclient: init 2024-03-25T18:06:50.539 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.538+0000 7f7bceac9640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:50.539 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.538+0000 7f7bceac9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:06:50.540 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.538+0000 7f7bceac9640 10 monclient(hunting): picked mon.noname-b con 0x7f7bc80a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:06:50.540 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.538+0000 7f7bceac9640 10 monclient(hunting): picked mon.noname-c con 0x7f7bc813f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:06:50.540 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.538+0000 7f7bceac9640 10 monclient(hunting): picked mon.noname-a con 0x7f7bc81430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:06:50.540 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.538+0000 7f7bceac9640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bc8142520 con 0x7f7bc81430c0 2024-03-25T18:06:50.540 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.538+0000 7f7bceac9640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bc80b28a0 con 0x7f7bc813f710 2024-03-25T18:06:50.540 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.538+0000 7f7bceac9640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bc80b2a30 con 0x7f7bc80a98f0 2024-03-25T18:06:50.540 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.538+0000 7f7bceac9640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:50.540 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.538+0000 7f7bceac9640 10 monclient(hunting): authenticate will time out at 7159.598364s 2024-03-25T18:06:50.546 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.545+0000 7f7bc7fff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f7bc813f710 0x7f7bc81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:41822/0 (socket says 172.21.15.66:41822) 2024-03-25T18:06:50.546 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.545+0000 7f7bc7fff640 1 -- 172.21.15.66:0/3188474322 learned_addr learned my addr 172.21.15.66:0/3188474322 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:06:50.562 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.561+0000 7f7bc77fe640 1 -- 172.21.15.66:0/3188474322 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 541479899 0 0) 0x7f7bc80b28a0 con 0x7f7bc813f710 2024-03-25T18:06:50.562 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.561+0000 7f7bc77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:06:50.562 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.561+0000 7f7bc77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:50.562 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.561+0000 7f7bc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:50.562 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.561+0000 7f7bc77fe640 10 monclient(hunting): my global_id is 4351 2024-03-25T18:06:50.562 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.561+0000 7f7bc77fe640 1 -- 172.21.15.66:0/3188474322 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bb8003520 con 0x7f7bc813f710 2024-03-25T18:06:50.567 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.566+0000 7f7bc77fe640 1 -- 172.21.15.66:0/3188474322 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 790549751 0 0) 0x7f7bc80b2a30 con 0x7f7bc80a98f0 2024-03-25T18:06:50.567 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.566+0000 7f7bc77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:06:50.567 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.566+0000 7f7bc77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:50.567 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.566+0000 7f7bc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:50.567 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.566+0000 7f7bc77fe640 10 monclient(hunting): my global_id is 4355 2024-03-25T18:06:50.568 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.566+0000 7f7bc77fe640 1 -- 172.21.15.66:0/3188474322 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bc80b28a0 con 0x7f7bc80a98f0 2024-03-25T18:06:50.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.570+0000 7f7bc77fe640 1 -- 172.21.15.66:0/3188474322 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2241863211 0 0) 0x7f7bc8142520 con 0x7f7bc81430c0 2024-03-25T18:06:50.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.570+0000 7f7bc77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:06:50.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.570+0000 7f7bc77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:50.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.570+0000 7f7bc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:50.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.570+0000 7f7bc77fe640 10 monclient(hunting): my global_id is 4356 2024-03-25T18:06:50.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.570+0000 7f7bc77fe640 1 -- 172.21.15.66:0/3188474322 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bc80b2a30 con 0x7f7bc81430c0 2024-03-25T18:06:50.580 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.578+0000 7f7bc77fe640 1 -- 172.21.15.66:0/3188474322 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1738611576 0 0) 0x7f7bc80b28a0 con 0x7f7bc80a98f0 2024-03-25T18:06:50.580 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.578+0000 7f7bc77fe640 1 -- 172.21.15.66:0/3188474322 >> v1:172.21.15.193:6789/0 conn(0x7f7bc813f710 legacy=0x7f7bc81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:50.580 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.578+0000 7f7bc77fe640 1 -- 172.21.15.66:0/3188474322 >> v1:172.21.15.66:6789/0 conn(0x7f7bc81430c0 legacy=0x7f7bc814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:50.580 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.578+0000 7f7bc77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:06:50.580 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.578+0000 7f7bc77fe640 1 monclient: found mon.noname-b 2024-03-25T18:06:50.580 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.578+0000 7f7bc77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:50.580 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.578+0000 7f7bc77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.66:6790/0 2024-03-25T18:06:50.580 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.578+0000 7f7bc77fe640 1 -- 172.21.15.66:0/3188474322 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc80b44c0 con 0x7f7bc80a98f0 2024-03-25T18:06:50.581 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.578+0000 7f7bc77fe640 10 monclient: _finish_auth 0 2024-03-25T18:06:50.581 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.578+0000 7f7bc77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:06:50.581 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.578+0000 7f7bc77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:50.581 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.578+0000 7f7bc77fe640 1 -- 172.21.15.66:0/3188474322 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7bbc001670 con 0x7f7bc80a98f0 2024-03-25T18:06:50.581 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.578+0000 7f7bceac9640 5 monclient: authenticate success, global_id 4355 2024-03-25T18:06:50.581 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.578+0000 7f7bceac9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:50.581 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.578+0000 7f7bc77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:50.581 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.578+0000 7f7bc77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:06:50.581 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.578+0000 7f7bc77fe640 10 monclient: dump: 2024-03-25T18:06:50.581 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:50.581 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:50.581 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:50.581 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:50.581 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:50.581 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:50.582 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:50.582 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:50.582 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:50.582 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:50.582 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.578+0000 7f7bceac9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:50.590 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.588+0000 7f7bc77fe640 1 -- 172.21.15.66:0/3188474322 <== mon.2 v1:172.21.15.66:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7bbc0017d0 con 0x7f7bc80a98f0 2024-03-25T18:06:50.590 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.588+0000 7f7bc77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:50.590 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.588+0000 7f7bc77fe640 1 -- 172.21.15.66:0/3188474322 <== mon.2 v1:172.21.15.66:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7bbc001be0 con 0x7f7bc80a98f0 2024-03-25T18:06:50.590 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.588+0000 7f7bceac9640 10 monclient: get_monmap_and_config success 2024-03-25T18:06:50.590 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.588+0000 7f7bceac9640 4 set_mon_vals no callback set 2024-03-25T18:06:50.590 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.588+0000 7f7bc77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:50.590 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.588+0000 7f7bc77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:06:50.590 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.588+0000 7f7bc77fe640 10 monclient: dump: 2024-03-25T18:06:50.591 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:50.591 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:50.591 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:50.591 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:50.591 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:50.591 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:50.591 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:50.591 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:50.591 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:50.591 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:50.594 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.589+0000 7f7bceac9640 10 monclient: shutdown 2024-03-25T18:06:50.594 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.589+0000 7f7bceac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:50.594 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.589+0000 7f7bceac9640 1 -- 172.21.15.66:0/3188474322 >> v1:172.21.15.66:6790/0 conn(0x7f7bc80a98f0 legacy=0x7f7bc80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:50.595 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.589+0000 7f7bceac9640 1 -- 172.21.15.66:0/3188474322 shutdown_connections 2024-03-25T18:06:50.595 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.589+0000 7f7bceac9640 1 -- 172.21.15.66:0/3188474322 >> 172.21.15.66:0/3188474322 conn(0x7f7bc80b7ea0 msgr2=0x7f7bc80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:50.595 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.590+0000 7f7bceac9640 1 -- 172.21.15.66:0/3188474322 shutdown_connections 2024-03-25T18:06:50.595 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.590+0000 7f7bceac9640 1 -- 172.21.15.66:0/3188474322 wait complete. 2024-03-25T18:06:50.595 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.590+0000 7f7bceac9640 10 monclient: build_initial_monmap 2024-03-25T18:06:50.595 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.590+0000 7f7bceac9640 10 monclient: monmap: 2024-03-25T18:06:50.595 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:06:50.595 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:06:50.595 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:06:50.595 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:06:50.595 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:50.595 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:06:50.595 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:06:50.595 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:06:50.596 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:06:50.596 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:50.596 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.591+0000 7f7bceac9640 1 librados: starting msgr at 2024-03-25T18:06:50.596 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.591+0000 7f7bceac9640 1 librados: starting objecter 2024-03-25T18:06:50.596 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.591+0000 7f7bceac9640 1 Processor -- start 2024-03-25T18:06:50.596 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.591+0000 7f7bceac9640 1 -- start start 2024-03-25T18:06:50.596 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.591+0000 7f7bceac9640 1 librados: setting wanted keys 2024-03-25T18:06:50.596 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.591+0000 7f7bceac9640 1 librados: calling monclient init 2024-03-25T18:06:50.596 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.591+0000 7f7bceac9640 10 monclient: init 2024-03-25T18:06:50.596 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.592+0000 7f7bceac9640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:50.596 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.592+0000 7f7bceac9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:06:50.596 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.592+0000 7f7bceac9640 10 monclient(hunting): picked mon.noname-b con 0x7f7bc80a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:06:50.596 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.592+0000 7f7bceac9640 10 monclient(hunting): picked mon.noname-a con 0x7f7bc813f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:06:50.596 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.592+0000 7f7bceac9640 10 monclient(hunting): picked mon.noname-c con 0x7f7bc81430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:06:50.596 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.592+0000 7f7bceac9640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bc8136e60 con 0x7f7bc813f710 2024-03-25T18:06:50.596 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.592+0000 7f7bceac9640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bc81381a0 con 0x7f7bc80a98f0 2024-03-25T18:06:50.597 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.592+0000 7f7bceac9640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bc81370b0 con 0x7f7bc81430c0 2024-03-25T18:06:50.597 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.592+0000 7f7bceac9640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:50.597 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.592+0000 7f7bceac9640 10 monclient(hunting): authenticate will time out at 7159.652329s 2024-03-25T18:06:50.599 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.598+0000 7f7bc7fff640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f7bc813f710 0x7f7bc813a470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:50838/0 (socket says 172.21.15.66:50838) 2024-03-25T18:06:50.599 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.598+0000 7f7bc7fff640 1 -- 172.21.15.66:0/158358840 learned_addr learned my addr 172.21.15.66:0/158358840 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:06:50.605 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.604+0000 7f7bc57fa640 10 client.?.objecter ms_handle_connect 0x7f7bc813f710 2024-03-25T18:06:50.605 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.604+0000 7f7bc57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:50.606 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.605+0000 7f7bc57fa640 10 client.?.objecter ms_handle_connect 0x7f7bc81430c0 2024-03-25T18:06:50.606 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.605+0000 7f7bc57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:50.612 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.611+0000 7f7bc57fa640 10 client.?.objecter ms_handle_connect 0x7f7bc80a98f0 2024-03-25T18:06:50.612 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.611+0000 7f7bc57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:50.616 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.615+0000 7f7bc57fa640 1 -- 172.21.15.66:0/158358840 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 239538668 0 0) 0x7f7bc81370b0 con 0x7f7bc81430c0 2024-03-25T18:06:50.616 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.615+0000 7f7bc57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:06:50.616 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.615+0000 7f7bc57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:50.616 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.615+0000 7f7bc57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:50.617 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.615+0000 7f7bc57fa640 10 monclient(hunting): my global_id is 4358 2024-03-25T18:06:50.617 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.615+0000 7f7bc57fa640 1 -- 172.21.15.66:0/158358840 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bac004eb0 con 0x7f7bc81430c0 2024-03-25T18:06:50.621 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.620+0000 7f7bc57fa640 1 -- 172.21.15.66:0/158358840 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3690967219 0 0) 0x7f7bc8136e60 con 0x7f7bc813f710 2024-03-25T18:06:50.621 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.620+0000 7f7bc57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:06:50.621 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.620+0000 7f7bc57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:50.621 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.620+0000 7f7bc57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:50.621 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.620+0000 7f7bc57fa640 10 monclient(hunting): my global_id is 4359 2024-03-25T18:06:50.621 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.620+0000 7f7bc57fa640 1 -- 172.21.15.66:0/158358840 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bc81370b0 con 0x7f7bc813f710 2024-03-25T18:06:50.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.622+0000 7f7bc57fa640 1 -- 172.21.15.66:0/158358840 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244905546 0 0) 0x7f7bc81381a0 con 0x7f7bc80a98f0 2024-03-25T18:06:50.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.622+0000 7f7bc57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:06:50.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.622+0000 7f7bc57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:50.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.622+0000 7f7bc57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:50.624 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.622+0000 7f7bc57fa640 10 monclient(hunting): my global_id is 4354 2024-03-25T18:06:50.624 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.622+0000 7f7bc57fa640 1 -- 172.21.15.66:0/158358840 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bc8136e60 con 0x7f7bc80a98f0 2024-03-25T18:06:50.634 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.632+0000 7f7bc57fa640 1 -- 172.21.15.66:0/158358840 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2671136799 0 0) 0x7f7bac004eb0 con 0x7f7bc81430c0 2024-03-25T18:06:50.634 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.632+0000 7f7bc57fa640 1 -- 172.21.15.66:0/158358840 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7bc81381a0 con 0x7f7bc81430c0 2024-03-25T18:06:50.634 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.632+0000 7f7bc57fa640 1 -- 172.21.15.66:0/158358840 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7bc0001bd0 con 0x7f7bc81430c0 2024-03-25T18:06:50.634 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.632+0000 7f7bc57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:50.634 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.632+0000 7f7bc57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:06:50.634 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.632+0000 7f7bc57fa640 10 monclient(hunting): dump: 2024-03-25T18:06:50.634 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:50.634 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:50.634 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:50.634 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:50.634 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:50.634 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:50.634 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:50.634 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:50.635 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:50.635 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:50.635 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.632+0000 7f7bc57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:06:50.635 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.633+0000 7f7bc57fa640 1 -- 172.21.15.66:0/158358840 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1765694486 0 0) 0x7f7bc81370b0 con 0x7f7bc813f710 2024-03-25T18:06:50.635 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.633+0000 7f7bc57fa640 1 -- 172.21.15.66:0/158358840 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7bac004eb0 con 0x7f7bc813f710 2024-03-25T18:06:50.635 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.633+0000 7f7bc57fa640 1 -- 172.21.15.66:0/158358840 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7bb40018d0 con 0x7f7bc813f710 2024-03-25T18:06:50.635 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.633+0000 7f7bc57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:50.636 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.633+0000 7f7bc57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:06:50.636 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.633+0000 7f7bc57fa640 10 monclient(hunting): dump: 2024-03-25T18:06:50.636 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:50.636 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:50.636 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:50.636 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:50.636 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:50.636 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:50.636 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:50.636 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:50.636 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:50.636 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:50.639 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.637+0000 7f7bc57fa640 1 -- 172.21.15.66:0/158358840 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1857415624 0 0) 0x7f7bc8136e60 con 0x7f7bc80a98f0 2024-03-25T18:06:50.639 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.637+0000 7f7bc57fa640 1 -- 172.21.15.66:0/158358840 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7bc81370b0 con 0x7f7bc80a98f0 2024-03-25T18:06:50.639 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.637+0000 7f7bc57fa640 1 -- 172.21.15.66:0/158358840 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7bbc001ae0 con 0x7f7bc80a98f0 2024-03-25T18:06:50.639 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.637+0000 7f7bc57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:50.639 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.637+0000 7f7bc57fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:50.639 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.637+0000 7f7bc57fa640 10 monclient(hunting): dump: 2024-03-25T18:06:50.639 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:50.639 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:50.639 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:50.639 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:50.639 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:50.639 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:50.639 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:50.640 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:50.640 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:50.640 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:50.647 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.645+0000 7f7bc57fa640 1 -- 172.21.15.66:0/158358840 <== mon.0 v1:172.21.15.66:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1568255064 0 0) 0x7f7bac004eb0 con 0x7f7bc813f710 2024-03-25T18:06:50.647 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.646+0000 7f7bc57fa640 1 -- 172.21.15.66:0/158358840 >> v1:172.21.15.66:6790/0 conn(0x7f7bc81430c0 legacy=0x7f7bc813ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:50.647 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.646+0000 7f7bc57fa640 1 -- 172.21.15.66:0/158358840 >> v1:172.21.15.193:6789/0 conn(0x7f7bc80a98f0 legacy=0x7f7bc8137a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:50.648 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.646+0000 7f7bc57fa640 10 monclient: _finish_hunting 0 2024-03-25T18:06:50.648 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.646+0000 7f7bc57fa640 1 monclient: found mon.a 2024-03-25T18:06:50.648 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.646+0000 7f7bc57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:50.648 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.646+0000 7f7bc57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:06:50.648 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.646+0000 7f7bc57fa640 1 -- 172.21.15.66:0/158358840 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc8137390 con 0x7f7bc813f710 2024-03-25T18:06:50.648 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.646+0000 7f7bc57fa640 10 monclient: _finish_auth 0 2024-03-25T18:06:50.648 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.646+0000 7f7bc57fa640 10 monclient: _check_auth_tickets 2024-03-25T18:06:50.648 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.646+0000 7f7bc57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:50.648 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.646+0000 7f7bceac9640 5 monclient: authenticate success, global_id 4359 2024-03-25T18:06:50.648 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.646+0000 7f7bceac9640 10 monclient: _renew_subs 2024-03-25T18:06:50.648 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.646+0000 7f7bceac9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:06:50.648 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.646+0000 7f7bceac9640 1 -- 172.21.15.66:0/158358840 --> v1:172.21.15.66:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bc81376d0 con 0x7f7bc813f710 2024-03-25T18:06:50.648 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.646+0000 7f7bceac9640 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:06:50.648 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.646+0000 7f7bceac9640 10 monclient: _renew_subs 2024-03-25T18:06:50.649 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.646+0000 7f7bceac9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:06:50.649 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.646+0000 7f7bceac9640 1 -- 172.21.15.66:0/158358840 --> v1:172.21.15.66:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bc8135530 con 0x7f7bc813f710 2024-03-25T18:06:50.649 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.646+0000 7f7bceac9640 1 librados: init done 2024-03-25T18:06:50.649 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.647+0000 7f7bceac9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:50.649 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.647+0000 7f7bceac9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:50.649 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.647+0000 7f7bceac9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:06:50.649 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.647+0000 7f7bceac9640 1 -- 172.21.15.66:0/158358840 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bc80b3ef0 con 0x7f7bc813f710 2024-03-25T18:06:50.992 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.991+0000 7f7bc57fa640 1 -- 172.21.15.66:0/158358840 <== mon.0 v1:172.21.15.66:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7bb4001d00 con 0x7f7bc813f710 2024-03-25T18:06:50.992 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.991+0000 7f7bc57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:50.992 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.991+0000 7f7bc57fa640 1 -- 172.21.15.66:0/158358840 <== mon.0 v1:172.21.15.66:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7bb4004980 con 0x7f7bc813f710 2024-03-25T18:06:50.993 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.991+0000 7f7bc57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:50.993 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.991+0000 7f7bc57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:06:50.993 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.991+0000 7f7bc57fa640 10 monclient: dump: 2024-03-25T18:06:50.993 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:50.993 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:50.993 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:50.993 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:50.993 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:50.993 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:50.993 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:50.993 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:50.993 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:50.993 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:50.993 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.991+0000 7f7bc6ffd640 4 set_mon_vals no callback set 2024-03-25T18:06:50.994 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.992+0000 7f7bc57fa640 1 -- 172.21.15.66:0/158358840 <== mon.0 v1:172.21.15.66:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f7bb401aa10 con 0x7f7bc813f710 2024-03-25T18:06:50.994 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.992+0000 7f7bcc865640 10 monclient: get_auth_request con 0x7f7bac06e110 auth_method 0 2024-03-25T18:06:50.994 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.993+0000 7f7bc57fa640 10 client.4359.objecter ms_handle_connect 0x7f7bac06e110 2024-03-25T18:06:51.002 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.999+0000 7f7bc57fa640 1 -- 172.21.15.66:0/158358840 <== mon.0 v1:172.21.15.66:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f7bb4054740 con 0x7f7bc813f710 2024-03-25T18:06:51.002 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.999+0000 7f7bc57fa640 10 client.4359.objecter ms_dispatch 0x7f7bc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:06:51.002 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.999+0000 7f7bc57fa640 3 client.4359.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:06:51.002 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.999+0000 7f7bc57fa640 3 client.4359.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:06:51.002 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.999+0000 7f7bc57fa640 20 client.4359.objecter dump_active .. 0 homeless 2024-03-25T18:06:51.002 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.999+0000 7f7bc57fa640 1 -- 172.21.15.66:0/158358840 <== mon.0 v1:172.21.15.66:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7bb4050080 con 0x7f7bc813f710 2024-03-25T18:06:51.002 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.999+0000 7f7bc57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:51.002 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:50.999+0000 7f7bc57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:06:51.262 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:51.261+0000 7f7bceac9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:51.263 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:51.261+0000 7f7bceac9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:51.263 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:51.261+0000 7f7bceac9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:06:51.263 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:51.261+0000 7f7bceac9640 1 -- 172.21.15.66:0/158358840 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7bc81357e0 con 0x7f7bc813f710 2024-03-25T18:06:51.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:51.295+0000 7f7bc57fa640 1 -- 172.21.15.66:0/158358840 <== mon.0 v1:172.21.15.66:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f7bb4052180 con 0x7f7bc813f710 2024-03-25T18:06:51.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:51.295+0000 7f7bc57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:51.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:51.295+0000 7f7bc57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:06:51.297 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:06:51.297 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:06:51.299 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:51.297+0000 7f7bceac9640 10 librados: watch_flush enter 2024-03-25T18:06:51.299 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:51.297+0000 7f7bceac9640 10 librados: watch_flush exit 2024-03-25T18:06:51.299 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:51.297+0000 7f7bceac9640 20 client.4359.objecter shutdown clearing up homeless session... 2024-03-25T18:06:51.299 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:51.297+0000 7f7bceac9640 10 client.4359.objecter successfully canceled tick 2024-03-25T18:06:51.299 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:51.297+0000 7f7bceac9640 1 -- 172.21.15.66:0/158358840 >> v1:172.21.15.193:6800/2275676803 conn(0x7f7bac06e110 legacy=0x7f7bac0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:51.299 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:51.297+0000 7f7bceac9640 10 monclient: shutdown 2024-03-25T18:06:51.299 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:51.297+0000 7f7bceac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:51.299 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:51.297+0000 7f7bceac9640 1 -- 172.21.15.66:0/158358840 >> v1:172.21.15.66:6789/0 conn(0x7f7bc813f710 legacy=0x7f7bc813a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:51.300 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:51.297+0000 7f7bceac9640 1 -- 172.21.15.66:0/158358840 shutdown_connections 2024-03-25T18:06:51.300 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:51.297+0000 7f7bceac9640 1 -- 172.21.15.66:0/158358840 >> 172.21.15.66:0/158358840 conn(0x7f7bc80b7ea0 msgr2=0x7f7bc8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:51.300 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:51.297+0000 7f7bceac9640 1 -- 172.21.15.66:0/158358840 shutdown_connections 2024-03-25T18:06:51.300 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:51.297+0000 7f7bceac9640 1 -- 172.21.15.66:0/158358840 wait complete. 2024-03-25T18:06:51.300 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:51.297+0000 7f7bceac9640 1 librados: shutdown 2024-03-25T18:06:51.311 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:06:58.117 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:06:58.236 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.235+0000 7f39b6fa3640 10 monclient: get_monmap_and_config 2024-03-25T18:06:58.237 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.235+0000 7f39b6fa3640 10 monclient: build_initial_monmap 2024-03-25T18:06:58.237 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.235+0000 7f39b6fa3640 10 monclient: monmap: 2024-03-25T18:06:58.237 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:06:58.237 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:58.237 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:06:58.236960+0000 2024-03-25T18:06:58.237 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:06:58.236960+0000 2024-03-25T18:06:58.237 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:58.237 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:58.237 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:06:58.237 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:06:58.237 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:06:58.237 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:58.238 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.236+0000 7f39b6fa3640 1 Processor -- start 2024-03-25T18:06:58.239 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.236+0000 7f39b6fa3640 1 -- start start 2024-03-25T18:06:58.239 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.236+0000 7f39b6fa3640 10 monclient: init 2024-03-25T18:06:58.239 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.237+0000 7f39b6fa3640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:58.239 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.237+0000 7f39b6fa3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:06:58.239 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.237+0000 7f39b6fa3640 10 monclient(hunting): picked mon.noname-c con 0x7f39b00a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:06:58.239 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.237+0000 7f39b6fa3640 10 monclient(hunting): picked mon.noname-a con 0x7f39b013f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:06:58.239 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.237+0000 7f39b6fa3640 10 monclient(hunting): picked mon.noname-b con 0x7f39b01430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:06:58.239 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.237+0000 7f39b6fa3640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b0142520 con 0x7f39b013f710 2024-03-25T18:06:58.239 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.237+0000 7f39b6fa3640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b00b28a0 con 0x7f39b00a98f0 2024-03-25T18:06:58.239 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.237+0000 7f39b6fa3640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b00b2a30 con 0x7f39b01430c0 2024-03-25T18:06:58.239 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.237+0000 7f39b6fa3640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:58.239 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.237+0000 7f39b6fa3640 10 monclient(hunting): authenticate will time out at 7167.297472s 2024-03-25T18:06:58.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.252+0000 7f39affff640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f39b013f710 0x7f39b01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:50562/0 (socket says 172.21.15.66:50562) 2024-03-25T18:06:58.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.252+0000 7f39affff640 1 -- 172.21.15.66:0/3585493318 learned_addr learned my addr 172.21.15.66:0/3585493318 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:06:58.266 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.265+0000 7f39af7fe640 1 -- 172.21.15.66:0/3585493318 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 42479217 0 0) 0x7f39b00b28a0 con 0x7f39b00a98f0 2024-03-25T18:06:58.266 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.265+0000 7f39af7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:06:58.266 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.265+0000 7f39af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:58.266 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.265+0000 7f39af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:58.266 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.265+0000 7f39af7fe640 10 monclient(hunting): my global_id is 4357 2024-03-25T18:06:58.266 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.265+0000 7f39af7fe640 1 -- 172.21.15.66:0/3585493318 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39a0003520 con 0x7f39b00a98f0 2024-03-25T18:06:58.267 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.266+0000 7f39af7fe640 1 -- 172.21.15.66:0/3585493318 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173265957 0 0) 0x7f39b00b2a30 con 0x7f39b01430c0 2024-03-25T18:06:58.267 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.266+0000 7f39af7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:06:58.267 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.266+0000 7f39af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:58.267 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.266+0000 7f39af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:58.267 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.266+0000 7f39af7fe640 10 monclient(hunting): my global_id is 4361 2024-03-25T18:06:58.267 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.266+0000 7f39af7fe640 1 -- 172.21.15.66:0/3585493318 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39b00b28a0 con 0x7f39b01430c0 2024-03-25T18:06:58.271 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.269+0000 7f39af7fe640 1 -- 172.21.15.66:0/3585493318 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 133039721 0 0) 0x7f39b0142520 con 0x7f39b013f710 2024-03-25T18:06:58.271 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.269+0000 7f39af7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:06:58.271 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.269+0000 7f39af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:58.271 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.269+0000 7f39af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:58.271 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.269+0000 7f39af7fe640 10 monclient(hunting): my global_id is 4362 2024-03-25T18:06:58.271 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.269+0000 7f39af7fe640 1 -- 172.21.15.66:0/3585493318 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39b00b2a30 con 0x7f39b013f710 2024-03-25T18:06:58.273 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.271+0000 7f39af7fe640 1 -- 172.21.15.66:0/3585493318 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2387096853 0 0) 0x7f39a0003520 con 0x7f39b00a98f0 2024-03-25T18:06:58.273 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.271+0000 7f39af7fe640 1 -- 172.21.15.66:0/3585493318 >> v1:172.21.15.66:6790/0 conn(0x7f39b01430c0 legacy=0x7f39b014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:58.273 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.271+0000 7f39af7fe640 1 -- 172.21.15.66:0/3585493318 >> v1:172.21.15.66:6789/0 conn(0x7f39b013f710 legacy=0x7f39b01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:58.273 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.271+0000 7f39af7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:06:58.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.271+0000 7f39af7fe640 1 monclient: found mon.noname-c 2024-03-25T18:06:58.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.271+0000 7f39af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:58.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.271+0000 7f39af7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:06:58.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.271+0000 7f39af7fe640 1 -- 172.21.15.66:0/3585493318 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b00b44c0 con 0x7f39b00a98f0 2024-03-25T18:06:58.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.271+0000 7f39af7fe640 10 monclient: _finish_auth 0 2024-03-25T18:06:58.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.271+0000 7f39af7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:06:58.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.271+0000 7f39af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:58.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.271+0000 7f39af7fe640 1 -- 172.21.15.66:0/3585493318 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f39a4001690 con 0x7f39b00a98f0 2024-03-25T18:06:58.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.271+0000 7f39af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:58.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.271+0000 7f39af7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:06:58.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.271+0000 7f39af7fe640 10 monclient: dump: 2024-03-25T18:06:58.274 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:58.274 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:58.274 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:58.275 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:58.275 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:58.275 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:58.275 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:58.275 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:58.275 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:58.275 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:58.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.271+0000 7f39b6fa3640 5 monclient: authenticate success, global_id 4357 2024-03-25T18:06:58.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.271+0000 7f39b6fa3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:06:58.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.294+0000 7f39af7fe640 1 -- 172.21.15.66:0/3585493318 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39a40017f0 con 0x7f39b00a98f0 2024-03-25T18:06:58.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.294+0000 7f39af7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:58.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.294+0000 7f39af7fe640 1 -- 172.21.15.66:0/3585493318 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f39a4001c00 con 0x7f39b00a98f0 2024-03-25T18:06:58.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.294+0000 7f39b6fa3640 10 monclient: get_monmap_and_config success 2024-03-25T18:06:58.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.294+0000 7f39b6fa3640 4 set_mon_vals no callback set 2024-03-25T18:06:58.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.294+0000 7f39af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:58.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.294+0000 7f39af7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:06:58.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.294+0000 7f39af7fe640 10 monclient: dump: 2024-03-25T18:06:58.296 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:58.296 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:58.296 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:58.296 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:58.296 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:58.296 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:58.296 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:58.297 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:58.297 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:58.297 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:58.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.295+0000 7f39b6fa3640 10 monclient: shutdown 2024-03-25T18:06:58.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.295+0000 7f39b6fa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:58.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.295+0000 7f39b6fa3640 1 -- 172.21.15.66:0/3585493318 >> v1:172.21.15.193:6789/0 conn(0x7f39b00a98f0 legacy=0x7f39b00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:58.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.295+0000 7f39b6fa3640 1 -- 172.21.15.66:0/3585493318 shutdown_connections 2024-03-25T18:06:58.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.295+0000 7f39b6fa3640 1 -- 172.21.15.66:0/3585493318 >> 172.21.15.66:0/3585493318 conn(0x7f39b00b7ea0 msgr2=0x7f39b00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:58.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.295+0000 7f39b6fa3640 1 -- 172.21.15.66:0/3585493318 shutdown_connections 2024-03-25T18:06:58.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.295+0000 7f39b6fa3640 1 -- 172.21.15.66:0/3585493318 wait complete. 2024-03-25T18:06:58.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.296+0000 7f39b6fa3640 10 monclient: build_initial_monmap 2024-03-25T18:06:58.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.296+0000 7f39b6fa3640 10 monclient: monmap: 2024-03-25T18:06:58.298 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:06:58.298 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:06:58.298 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:06:58.299 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:06:58.299 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:06:58.299 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:06:58.299 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:06:58.299 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:06:58.299 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:06:58.299 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:58.300 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.297+0000 7f39b6fa3640 1 librados: starting msgr at 2024-03-25T18:06:58.300 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.297+0000 7f39b6fa3640 1 librados: starting objecter 2024-03-25T18:06:58.300 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.297+0000 7f39b6fa3640 1 Processor -- start 2024-03-25T18:06:58.300 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.297+0000 7f39b6fa3640 1 -- start start 2024-03-25T18:06:58.300 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.297+0000 7f39b6fa3640 1 librados: setting wanted keys 2024-03-25T18:06:58.301 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.297+0000 7f39b6fa3640 1 librados: calling monclient init 2024-03-25T18:06:58.301 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.297+0000 7f39b6fa3640 10 monclient: init 2024-03-25T18:06:58.301 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.298+0000 7f39b6fa3640 10 monclient: _reopen_session rank -1 2024-03-25T18:06:58.301 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.298+0000 7f39b6fa3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:06:58.301 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.298+0000 7f39b6fa3640 10 monclient(hunting): picked mon.noname-c con 0x7f39b00a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:06:58.301 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.298+0000 7f39b6fa3640 10 monclient(hunting): picked mon.noname-b con 0x7f39b013f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:06:58.301 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.298+0000 7f39b6fa3640 10 monclient(hunting): picked mon.noname-a con 0x7f39b01430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:06:58.301 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.298+0000 7f39b6fa3640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b0138b40 con 0x7f39b01430c0 2024-03-25T18:06:58.301 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.298+0000 7f39b6fa3640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b0138e20 con 0x7f39b013f710 2024-03-25T18:06:58.301 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.298+0000 7f39b6fa3640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b0139100 con 0x7f39b00a98f0 2024-03-25T18:06:58.301 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.298+0000 7f39b6fa3640 10 monclient(hunting): _renew_subs 2024-03-25T18:06:58.301 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.298+0000 7f39b6fa3640 10 monclient(hunting): authenticate will time out at 7167.358537s 2024-03-25T18:06:58.304 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.303+0000 7f39b5540640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f39b01430c0 0x7f39b01e1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:50618/0 (socket says 172.21.15.66:50618) 2024-03-25T18:06:58.304 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.303+0000 7f39b5540640 1 -- 172.21.15.66:0/519469426 learned_addr learned my addr 172.21.15.66:0/519469426 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:06:58.305 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.303+0000 7f39ad7fa640 10 client.?.objecter ms_handle_connect 0x7f39b01430c0 2024-03-25T18:06:58.305 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.303+0000 7f39ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:58.305 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.304+0000 7f39ad7fa640 10 client.?.objecter ms_handle_connect 0x7f39b00a98f0 2024-03-25T18:06:58.305 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.304+0000 7f39ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:58.316 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.314+0000 7f39ad7fa640 1 -- 172.21.15.66:0/519469426 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2291300710 0 0) 0x7f39b0138b40 con 0x7f39b01430c0 2024-03-25T18:06:58.316 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.314+0000 7f39ad7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:06:58.316 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.314+0000 7f39ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:06:58.316 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.314+0000 7f39ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:06:58.316 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.314+0000 7f39ad7fa640 10 monclient(hunting): my global_id is 4365 2024-03-25T18:06:58.316 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.314+0000 7f39ad7fa640 1 -- 172.21.15.66:0/519469426 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3994004eb0 con 0x7f39b01430c0 2024-03-25T18:06:58.325 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.324+0000 7f39ad7fa640 1 -- 172.21.15.66:0/519469426 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4037118400 0 0) 0x7f3994004eb0 con 0x7f39b01430c0 2024-03-25T18:06:58.325 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.324+0000 7f39ad7fa640 1 -- 172.21.15.66:0/519469426 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39b0138b40 con 0x7f39b01430c0 2024-03-25T18:06:58.325 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.324+0000 7f39ad7fa640 1 -- 172.21.15.66:0/519469426 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f39a8001c90 con 0x7f39b01430c0 2024-03-25T18:06:58.325 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.324+0000 7f39ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:58.326 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.324+0000 7f39ad7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:06:58.326 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.324+0000 7f39ad7fa640 10 monclient(hunting): dump: 2024-03-25T18:06:58.326 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:58.326 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:58.326 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:58.326 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:58.326 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:58.326 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:58.326 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:58.326 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:58.326 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:58.326 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:58.326 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.324+0000 7f39ad7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:06:58.327 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.326+0000 7f39ad7fa640 10 client.?.objecter ms_handle_connect 0x7f39b013f710 2024-03-25T18:06:58.327 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.326+0000 7f39ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:06:58.334 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.333+0000 7f39ad7fa640 1 -- 172.21.15.66:0/519469426 <== mon.0 v1:172.21.15.66:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3406536536 0 0) 0x7f39b0138b40 con 0x7f39b01430c0 2024-03-25T18:06:58.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.333+0000 7f39ad7fa640 1 -- 172.21.15.66:0/519469426 >> v1:172.21.15.66:6790/0 conn(0x7f39b00a98f0 legacy=0x7f39b0137380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:58.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.333+0000 7f39ad7fa640 1 -- 172.21.15.66:0/519469426 >> v1:172.21.15.193:6789/0 conn(0x7f39b013f710 legacy=0x7f39b0137a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:58.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.333+0000 7f39ad7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:06:58.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.333+0000 7f39ad7fa640 1 monclient: found mon.a 2024-03-25T18:06:58.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.333+0000 7f39ad7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:06:58.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.333+0000 7f39ad7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:06:58.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.333+0000 7f39ad7fa640 1 -- 172.21.15.66:0/519469426 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b01397a0 con 0x7f39b01430c0 2024-03-25T18:06:58.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.333+0000 7f39ad7fa640 10 monclient: _finish_auth 0 2024-03-25T18:06:58.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.333+0000 7f39ad7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:06:58.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.333+0000 7f39ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:06:58.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.333+0000 7f39b6fa3640 5 monclient: authenticate success, global_id 4365 2024-03-25T18:06:58.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.333+0000 7f39b6fa3640 10 monclient: _renew_subs 2024-03-25T18:06:58.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.333+0000 7f39b6fa3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:06:58.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.333+0000 7f39b6fa3640 1 -- 172.21.15.66:0/519469426 --> v1:172.21.15.66:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b0136aa0 con 0x7f39b01430c0 2024-03-25T18:06:58.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.333+0000 7f39b6fa3640 10 client.4365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:06:58.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.333+0000 7f39b6fa3640 10 monclient: _renew_subs 2024-03-25T18:06:58.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.333+0000 7f39b6fa3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:06:58.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.333+0000 7f39b6fa3640 1 -- 172.21.15.66:0/519469426 --> v1:172.21.15.66:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b013c320 con 0x7f39b01430c0 2024-03-25T18:06:58.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.333+0000 7f39b6fa3640 1 librados: init done 2024-03-25T18:06:58.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.334+0000 7f39b6fa3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:58.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.334+0000 7f39b6fa3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:58.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.334+0000 7f39b6fa3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:06:58.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.334+0000 7f39b6fa3640 1 -- 172.21.15.66:0/519469426 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b01395b0 con 0x7f39b01430c0 2024-03-25T18:06:58.653 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.652+0000 7f39ad7fa640 1 -- 172.21.15.66:0/519469426 <== mon.0 v1:172.21.15.66:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39a8003970 con 0x7f39b01430c0 2024-03-25T18:06:58.653 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.652+0000 7f39ad7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:06:58.653 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.652+0000 7f39ad7fa640 1 -- 172.21.15.66:0/519469426 <== mon.0 v1:172.21.15.66:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f39a8001df0 con 0x7f39b01430c0 2024-03-25T18:06:58.653 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.652+0000 7f39ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:06:58.653 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.652+0000 7f39ad7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:06:58.653 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.652+0000 7f39ad7fa640 10 monclient: dump: 2024-03-25T18:06:58.654 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:06:58.654 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:06:58.654 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:58.654 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:06:58.654 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:06:58.654 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:06:58.654 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:06:58.654 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:06:58.654 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:06:58.654 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:06:58.654 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.652+0000 7f39aeffd640 4 set_mon_vals no callback set 2024-03-25T18:06:58.655 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.652+0000 7f39ad7fa640 1 -- 172.21.15.66:0/519469426 <== mon.0 v1:172.21.15.66:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f39a8003e00 con 0x7f39b01430c0 2024-03-25T18:06:58.655 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.653+0000 7f39b4d3f640 10 monclient: get_auth_request con 0x7f399406dd90 auth_method 0 2024-03-25T18:06:58.655 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.653+0000 7f39ad7fa640 10 client.4365.objecter ms_handle_connect 0x7f399406dd90 2024-03-25T18:06:58.661 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.660+0000 7f39ad7fa640 1 -- 172.21.15.66:0/519469426 <== mon.0 v1:172.21.15.66:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f39a8054a90 con 0x7f39b01430c0 2024-03-25T18:06:58.661 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.660+0000 7f39ad7fa640 10 client.4365.objecter ms_dispatch 0x7f39b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:06:58.661 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.660+0000 7f39ad7fa640 3 client.4365.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:06:58.661 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.660+0000 7f39ad7fa640 3 client.4365.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:06:58.661 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.660+0000 7f39ad7fa640 20 client.4365.objecter dump_active .. 0 homeless 2024-03-25T18:06:58.661 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.660+0000 7f39ad7fa640 1 -- 172.21.15.66:0/519469426 <== mon.0 v1:172.21.15.66:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f39a80505a0 con 0x7f39b01430c0 2024-03-25T18:06:58.661 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.660+0000 7f39ad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:06:58.661 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.660+0000 7f39ad7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:06:58.927 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.925+0000 7f39b6fa3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:58.927 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.925+0000 7f39b6fa3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:58.927 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.925+0000 7f39b6fa3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:06:58.927 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:58.925+0000 7f39b6fa3640 1 -- 172.21.15.66:0/519469426 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39b00627b0 con 0x7f39b01430c0 2024-03-25T18:06:59.003 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:59.002+0000 7f39ad7fa640 1 -- 172.21.15.66:0/519469426 <== mon.0 v1:172.21.15.66:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f39a80524d0 con 0x7f39b01430c0 2024-03-25T18:06:59.003 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:59.002+0000 7f39ad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:06:59.003 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:59.002+0000 7f39ad7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:06:59.003 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:06:59.003 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:06:59.006 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:59.004+0000 7f39b6fa3640 10 librados: watch_flush enter 2024-03-25T18:06:59.006 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:59.004+0000 7f39b6fa3640 10 librados: watch_flush exit 2024-03-25T18:06:59.006 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:59.004+0000 7f39b6fa3640 20 client.4365.objecter shutdown clearing up homeless session... 2024-03-25T18:06:59.006 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:59.004+0000 7f39b6fa3640 10 client.4365.objecter successfully canceled tick 2024-03-25T18:06:59.006 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:59.004+0000 7f39b6fa3640 1 -- 172.21.15.66:0/519469426 >> v1:172.21.15.193:6800/2275676803 conn(0x7f399406dd90 legacy=0x7f3994070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:59.006 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:59.004+0000 7f39b6fa3640 10 monclient: shutdown 2024-03-25T18:06:59.006 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:59.004+0000 7f39b6fa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:06:59.006 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:59.004+0000 7f39b6fa3640 1 -- 172.21.15.66:0/519469426 >> v1:172.21.15.66:6789/0 conn(0x7f39b01430c0 legacy=0x7f39b01e1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:06:59.006 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:59.004+0000 7f39b6fa3640 1 -- 172.21.15.66:0/519469426 shutdown_connections 2024-03-25T18:06:59.006 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:59.004+0000 7f39b6fa3640 1 -- 172.21.15.66:0/519469426 >> 172.21.15.66:0/519469426 conn(0x7f39b00b7ea0 msgr2=0x7f39b014ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:06:59.006 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:59.004+0000 7f39b6fa3640 1 -- 172.21.15.66:0/519469426 shutdown_connections 2024-03-25T18:06:59.006 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:59.004+0000 7f39b6fa3640 1 -- 172.21.15.66:0/519469426 wait complete. 2024-03-25T18:06:59.007 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:06:59.004+0000 7f39b6fa3640 1 librados: shutdown 2024-03-25T18:06:59.017 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:07:04.018 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:03:34.998 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:04.067 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:03:35.048 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:04.108 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:03:35.089 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:04.133 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:03:35.113 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:04.139 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:03:35.097 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:04.143 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:03:35.102 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:04.149 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:03:35.129 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:04.172 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:03:35.131 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:04.182 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:03:35.163 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:05.824 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:07:05.943 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.942+0000 7fd95be51640 10 monclient: get_monmap_and_config 2024-03-25T18:07:05.943 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.942+0000 7fd95be51640 10 monclient: build_initial_monmap 2024-03-25T18:07:05.944 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.942+0000 7fd95be51640 10 monclient: monmap: 2024-03-25T18:07:05.944 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:07:05.944 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:05.944 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:07:05.943955+0000 2024-03-25T18:07:05.944 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:07:05.943955+0000 2024-03-25T18:07:05.944 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:05.944 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:05.944 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:07:05.944 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:07:05.944 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:07:05.944 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:05.945 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.943+0000 7fd95be51640 1 Processor -- start 2024-03-25T18:07:05.945 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.943+0000 7fd95be51640 1 -- start start 2024-03-25T18:07:05.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.943+0000 7fd95be51640 10 monclient: init 2024-03-25T18:07:05.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.944+0000 7fd95be51640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:05.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.944+0000 7fd95be51640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:07:05.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.944+0000 7fd95be51640 10 monclient(hunting): picked mon.noname-c con 0x7fd9540a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:07:05.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.944+0000 7fd95be51640 10 monclient(hunting): picked mon.noname-a con 0x7fd95413f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:07:05.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.944+0000 7fd95be51640 10 monclient(hunting): picked mon.noname-b con 0x7fd9541430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:07:05.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.944+0000 7fd95be51640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd954142520 con 0x7fd95413f710 2024-03-25T18:07:05.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.944+0000 7fd95be51640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9540b28a0 con 0x7fd9540a98f0 2024-03-25T18:07:05.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.944+0000 7fd95be51640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9540b2a30 con 0x7fd9541430c0 2024-03-25T18:07:05.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.944+0000 7fd95be51640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:05.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.944+0000 7fd95be51640 10 monclient(hunting): authenticate will time out at 7175.004547s 2024-03-25T18:07:05.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.954+0000 7fd95a3ee640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7fd9541430c0 0x7fd95414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:43938/0 (socket says 172.21.15.66:43938) 2024-03-25T18:07:05.956 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.954+0000 7fd95a3ee640 1 -- 172.21.15.66:0/2182265044 learned_addr learned my addr 172.21.15.66:0/2182265044 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:07:05.969 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.968+0000 7fd958beb640 1 -- 172.21.15.66:0/2182265044 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1322749569 0 0) 0x7fd9540b28a0 con 0x7fd9540a98f0 2024-03-25T18:07:05.969 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.968+0000 7fd958beb640 10 monclient(hunting): _negotiate 2024-03-25T18:07:05.969 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.968+0000 7fd958beb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:05.969 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.968+0000 7fd958beb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:05.969 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.968+0000 7fd958beb640 10 monclient(hunting): my global_id is 4363 2024-03-25T18:07:05.969 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.968+0000 7fd958beb640 1 -- 172.21.15.66:0/2182265044 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd940003520 con 0x7fd9540a98f0 2024-03-25T18:07:05.972 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.971+0000 7fd958beb640 1 -- 172.21.15.66:0/2182265044 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2950750308 0 0) 0x7fd9540b2a30 con 0x7fd9541430c0 2024-03-25T18:07:05.972 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.971+0000 7fd958beb640 10 monclient(hunting): _negotiate 2024-03-25T18:07:05.973 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.971+0000 7fd958beb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:05.973 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.971+0000 7fd958beb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:05.973 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.971+0000 7fd958beb640 10 monclient(hunting): my global_id is 4367 2024-03-25T18:07:05.973 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.971+0000 7fd958beb640 1 -- 172.21.15.66:0/2182265044 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9540b28a0 con 0x7fd9541430c0 2024-03-25T18:07:05.979 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.977+0000 7fd958beb640 1 -- 172.21.15.66:0/2182265044 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2310634551 0 0) 0x7fd9540b28a0 con 0x7fd9541430c0 2024-03-25T18:07:05.979 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.977+0000 7fd958beb640 1 -- 172.21.15.66:0/2182265044 >> v1:172.21.15.193:6789/0 conn(0x7fd9540a98f0 legacy=0x7fd9540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:05.979 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.977+0000 7fd958beb640 1 -- 172.21.15.66:0/2182265044 >> v1:172.21.15.66:6789/0 conn(0x7fd95413f710 legacy=0x7fd9541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:05.979 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.977+0000 7fd958beb640 10 monclient: _finish_hunting 0 2024-03-25T18:07:05.979 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.977+0000 7fd958beb640 1 monclient: found mon.noname-b 2024-03-25T18:07:05.979 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.977+0000 7fd958beb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:05.979 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.977+0000 7fd958beb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.66:6790/0 2024-03-25T18:07:05.979 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.977+0000 7fd958beb640 1 -- 172.21.15.66:0/2182265044 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9540b44c0 con 0x7fd9541430c0 2024-03-25T18:07:05.979 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.977+0000 7fd958beb640 10 monclient: _finish_auth 0 2024-03-25T18:07:05.979 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.977+0000 7fd958beb640 10 monclient: _check_auth_tickets 2024-03-25T18:07:05.979 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.977+0000 7fd958beb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:05.979 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.977+0000 7fd958beb640 1 -- 172.21.15.66:0/2182265044 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fd94c0019f0 con 0x7fd9541430c0 2024-03-25T18:07:05.979 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.977+0000 7fd958beb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:05.979 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.977+0000 7fd958beb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:07:05.980 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.977+0000 7fd958beb640 10 monclient: dump: 2024-03-25T18:07:05.980 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:05.980 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:05.980 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:05.980 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:05.980 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:05.980 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:05.980 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:05.980 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:05.980 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:05.980 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:05.980 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.977+0000 7fd95be51640 5 monclient: authenticate success, global_id 4367 2024-03-25T18:07:05.980 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.977+0000 7fd95be51640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:05.982 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.981+0000 7fd958beb640 1 -- 172.21.15.66:0/2182265044 <== mon.2 v1:172.21.15.66:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd94c001c10 con 0x7fd9541430c0 2024-03-25T18:07:05.982 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.981+0000 7fd958beb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:05.983 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.981+0000 7fd958beb640 1 -- 172.21.15.66:0/2182265044 <== mon.2 v1:172.21.15.66:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fd94c001f20 con 0x7fd9541430c0 2024-03-25T18:07:05.983 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.981+0000 7fd95be51640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:05.983 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.981+0000 7fd95be51640 4 set_mon_vals no callback set 2024-03-25T18:07:05.983 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.981+0000 7fd958beb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:05.983 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.981+0000 7fd958beb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:05.983 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.981+0000 7fd958beb640 10 monclient: dump: 2024-03-25T18:07:05.983 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:05.983 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:05.983 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:05.983 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:05.983 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:05.983 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:05.983 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:05.984 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:05.984 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:05.984 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:05.986 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.982+0000 7fd95be51640 10 monclient: shutdown 2024-03-25T18:07:05.986 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.982+0000 7fd95be51640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:05.986 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.982+0000 7fd95be51640 1 -- 172.21.15.66:0/2182265044 >> v1:172.21.15.66:6790/0 conn(0x7fd9541430c0 legacy=0x7fd95414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:05.986 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.982+0000 7fd95be51640 1 -- 172.21.15.66:0/2182265044 shutdown_connections 2024-03-25T18:07:05.986 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.982+0000 7fd95be51640 1 -- 172.21.15.66:0/2182265044 >> 172.21.15.66:0/2182265044 conn(0x7fd9540b7ea0 msgr2=0x7fd9540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:05.987 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.982+0000 7fd95be51640 1 -- 172.21.15.66:0/2182265044 shutdown_connections 2024-03-25T18:07:05.987 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.982+0000 7fd95be51640 1 -- 172.21.15.66:0/2182265044 wait complete. 2024-03-25T18:07:05.987 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.983+0000 7fd95be51640 10 monclient: build_initial_monmap 2024-03-25T18:07:05.987 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.983+0000 7fd95be51640 10 monclient: monmap: 2024-03-25T18:07:05.987 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:07:05.987 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:05.987 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:07:05.987 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:07:05.987 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:05.987 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:07:05.987 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:07:05.987 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:07:05.987 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:07:05.987 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:05.988 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.983+0000 7fd95be51640 1 librados: starting msgr at 2024-03-25T18:07:05.988 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.983+0000 7fd95be51640 1 librados: starting objecter 2024-03-25T18:07:05.988 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.983+0000 7fd95be51640 1 Processor -- start 2024-03-25T18:07:05.988 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.983+0000 7fd95be51640 1 -- start start 2024-03-25T18:07:05.988 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.983+0000 7fd95be51640 1 librados: setting wanted keys 2024-03-25T18:07:05.988 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.983+0000 7fd95be51640 1 librados: calling monclient init 2024-03-25T18:07:05.988 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.983+0000 7fd95be51640 10 monclient: init 2024-03-25T18:07:05.988 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.984+0000 7fd95be51640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:05.988 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.984+0000 7fd95be51640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:07:05.988 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.984+0000 7fd95be51640 10 monclient(hunting): picked mon.noname-a con 0x7fd9540a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:07:05.988 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.984+0000 7fd95be51640 10 monclient(hunting): picked mon.noname-b con 0x7fd95413f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:07:05.988 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.984+0000 7fd95be51640 10 monclient(hunting): picked mon.noname-c con 0x7fd9541430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:07:05.988 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.984+0000 7fd95be51640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9540ae940 con 0x7fd9540a98f0 2024-03-25T18:07:05.988 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.984+0000 7fd95be51640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9540b0870 con 0x7fd95413f710 2024-03-25T18:07:05.988 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.984+0000 7fd95be51640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9540b0b50 con 0x7fd9541430c0 2024-03-25T18:07:05.989 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.984+0000 7fd95be51640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:05.989 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.984+0000 7fd95be51640 10 monclient(hunting): authenticate will time out at 7175.045024s 2024-03-25T18:07:05.990 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.989+0000 7fd9593ec640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fd95413f710 0x7fd9541d7490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:35316/0 (socket says 172.21.15.66:35316) 2024-03-25T18:07:05.990 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.989+0000 7fd9593ec640 1 -- 172.21.15.66:0/853283543 learned_addr learned my addr 172.21.15.66:0/853283543 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:07:05.995 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.993+0000 7fd94a7fc640 10 client.?.objecter ms_handle_connect 0x7fd95413f710 2024-03-25T18:07:05.995 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:05.993+0000 7fd94a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:06.003 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.001+0000 7fd94a7fc640 1 -- 172.21.15.66:0/853283543 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797762061 0 0) 0x7fd9540b0870 con 0x7fd95413f710 2024-03-25T18:07:06.003 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.001+0000 7fd94a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:07:06.003 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.001+0000 7fd94a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:06.003 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.001+0000 7fd94a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:06.003 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.001+0000 7fd94a7fc640 10 monclient(hunting): my global_id is 4366 2024-03-25T18:07:06.003 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.001+0000 7fd94a7fc640 1 -- 172.21.15.66:0/853283543 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd938004eb0 con 0x7fd95413f710 2024-03-25T18:07:06.005 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.004+0000 7fd94a7fc640 10 client.?.objecter ms_handle_connect 0x7fd9540a98f0 2024-03-25T18:07:06.005 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.004+0000 7fd94a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:06.021 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.019+0000 7fd94a7fc640 1 -- 172.21.15.66:0/853283543 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 759686189 0 0) 0x7fd9540ae940 con 0x7fd9540a98f0 2024-03-25T18:07:06.021 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.020+0000 7fd94a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:07:06.021 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.020+0000 7fd94a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:06.021 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.020+0000 7fd94a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:06.021 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.020+0000 7fd94a7fc640 10 monclient(hunting): my global_id is 4371 2024-03-25T18:07:06.021 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.020+0000 7fd94a7fc640 1 -- 172.21.15.66:0/853283543 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9540b0870 con 0x7fd9540a98f0 2024-03-25T18:07:06.029 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.027+0000 7fd94a7fc640 1 -- 172.21.15.66:0/853283543 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 391199908 0 0) 0x7fd938004eb0 con 0x7fd95413f710 2024-03-25T18:07:06.029 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.028+0000 7fd94a7fc640 1 -- 172.21.15.66:0/853283543 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9540ae940 con 0x7fd95413f710 2024-03-25T18:07:06.029 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.028+0000 7fd94a7fc640 1 -- 172.21.15.66:0/853283543 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fd9440019d0 con 0x7fd95413f710 2024-03-25T18:07:06.029 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.028+0000 7fd94a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:06.029 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.028+0000 7fd94a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:07:06.029 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.028+0000 7fd94a7fc640 10 monclient(hunting): dump: 2024-03-25T18:07:06.029 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:06.029 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:06.029 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:06.030 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:06.030 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:06.030 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:06.030 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:06.030 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:06.030 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:06.030 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:06.030 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.028+0000 7fd94a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:06.032 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.030+0000 7fd94a7fc640 10 client.?.objecter ms_handle_connect 0x7fd9541430c0 2024-03-25T18:07:06.032 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.030+0000 7fd94a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:06.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.039+0000 7fd94a7fc640 1 -- 172.21.15.66:0/853283543 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1697821448 0 0) 0x7fd9540ae940 con 0x7fd95413f710 2024-03-25T18:07:06.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.039+0000 7fd94a7fc640 1 -- 172.21.15.66:0/853283543 >> v1:172.21.15.66:6790/0 conn(0x7fd9541430c0 legacy=0x7fd9541d7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:06.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.039+0000 7fd94a7fc640 1 -- 172.21.15.66:0/853283543 >> v1:172.21.15.66:6789/0 conn(0x7fd9540a98f0 legacy=0x7fd9540af0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:06.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.039+0000 7fd94a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:07:06.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.039+0000 7fd94a7fc640 1 monclient: found mon.b 2024-03-25T18:07:06.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.039+0000 7fd94a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:06.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.039+0000 7fd94a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:07:06.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.039+0000 7fd94a7fc640 1 -- 172.21.15.66:0/853283543 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9540aeb70 con 0x7fd95413f710 2024-03-25T18:07:06.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.039+0000 7fd94a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:07:06.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.039+0000 7fd94a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:07:06.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.039+0000 7fd94a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:06.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.039+0000 7fd95be51640 5 monclient: authenticate success, global_id 4366 2024-03-25T18:07:06.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.039+0000 7fd95be51640 10 monclient: _renew_subs 2024-03-25T18:07:06.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.039+0000 7fd95be51640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:07:06.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.039+0000 7fd95be51640 1 -- 172.21.15.66:0/853283543 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9541d82b0 con 0x7fd95413f710 2024-03-25T18:07:06.042 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.039+0000 7fd95be51640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:06.042 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.039+0000 7fd95be51640 10 monclient: _renew_subs 2024-03-25T18:07:06.042 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.039+0000 7fd95be51640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:07:06.042 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.039+0000 7fd95be51640 1 -- 172.21.15.66:0/853283543 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9540af9d0 con 0x7fd95413f710 2024-03-25T18:07:06.042 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.039+0000 7fd95be51640 1 librados: init done 2024-03-25T18:07:06.042 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.040+0000 7fd95be51640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:06.042 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.040+0000 7fd95be51640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:06.043 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.040+0000 7fd95be51640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:07:06.043 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.040+0000 7fd95be51640 1 -- 172.21.15.66:0/853283543 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd95414f2d0 con 0x7fd95413f710 2024-03-25T18:07:06.314 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.312+0000 7fd94a7fc640 1 -- 172.21.15.66:0/853283543 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9440035f0 con 0x7fd95413f710 2024-03-25T18:07:06.314 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.312+0000 7fd94a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:06.314 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.312+0000 7fd94a7fc640 1 -- 172.21.15.66:0/853283543 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fd944003980 con 0x7fd95413f710 2024-03-25T18:07:06.314 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.312+0000 7fd94a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:06.314 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.312+0000 7fd94a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:06.314 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.312+0000 7fd94a7fc640 10 monclient: dump: 2024-03-25T18:07:06.315 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:06.315 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:06.315 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:06.315 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:06.315 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:06.315 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:06.315 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:06.315 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:06.315 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:06.315 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:06.315 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.312+0000 7fd94bfff640 4 set_mon_vals no callback set 2024-03-25T18:07:06.316 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.313+0000 7fd94a7fc640 1 -- 172.21.15.66:0/853283543 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7fd944003ba0 con 0x7fd95413f710 2024-03-25T18:07:06.316 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.314+0000 7fd959bed640 10 monclient: get_auth_request con 0x7fd93806dda0 auth_method 0 2024-03-25T18:07:06.316 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.315+0000 7fd94a7fc640 10 client.4366.objecter ms_handle_connect 0x7fd93806dda0 2024-03-25T18:07:06.321 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.319+0000 7fd94a7fc640 1 -- 172.21.15.66:0/853283543 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7fd944054760 con 0x7fd95413f710 2024-03-25T18:07:06.321 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.319+0000 7fd94a7fc640 10 client.4366.objecter ms_dispatch 0x7fd954000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:07:06.321 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.319+0000 7fd94a7fc640 3 client.4366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:07:06.321 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.319+0000 7fd94a7fc640 3 client.4366.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:07:06.321 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.319+0000 7fd94a7fc640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-25T18:07:06.321 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.320+0000 7fd94a7fc640 1 -- 172.21.15.66:0/853283543 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd944050270 con 0x7fd95413f710 2024-03-25T18:07:06.321 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.320+0000 7fd94a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:06.321 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.320+0000 7fd94a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:06.547 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.546+0000 7fd95be51640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:06.547 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.546+0000 7fd95be51640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:06.547 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.546+0000 7fd95be51640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:07:06.547 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.546+0000 7fd95be51640 1 -- 172.21.15.66:0/853283543 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9540aed00 con 0x7fd95413f710 2024-03-25T18:07:06.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.573+0000 7fd94a7fc640 1 -- 172.21.15.66:0/853283543 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7fd9440521a0 con 0x7fd95413f710 2024-03-25T18:07:06.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.573+0000 7fd94a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:06.575 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.573+0000 7fd94a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:06.575 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:07:06.575 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:07:06.577 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.575+0000 7fd95be51640 10 librados: watch_flush enter 2024-03-25T18:07:06.577 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.575+0000 7fd95be51640 10 librados: watch_flush exit 2024-03-25T18:07:06.577 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.575+0000 7fd95be51640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-25T18:07:06.578 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.575+0000 7fd95be51640 10 client.4366.objecter successfully canceled tick 2024-03-25T18:07:06.578 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.575+0000 7fd95be51640 1 -- 172.21.15.66:0/853283543 >> v1:172.21.15.193:6800/2275676803 conn(0x7fd93806dda0 legacy=0x7fd938070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:06.578 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.575+0000 7fd95be51640 10 monclient: shutdown 2024-03-25T18:07:06.578 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.575+0000 7fd95be51640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:06.578 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.575+0000 7fd95be51640 1 -- 172.21.15.66:0/853283543 >> v1:172.21.15.193:6789/0 conn(0x7fd95413f710 legacy=0x7fd9541d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:06.578 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.575+0000 7fd95be51640 1 -- 172.21.15.66:0/853283543 shutdown_connections 2024-03-25T18:07:06.578 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.575+0000 7fd95be51640 1 -- 172.21.15.66:0/853283543 >> 172.21.15.66:0/853283543 conn(0x7fd9540b7ea0 msgr2=0x7fd954074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:06.578 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.576+0000 7fd95be51640 1 -- 172.21.15.66:0/853283543 shutdown_connections 2024-03-25T18:07:06.578 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.576+0000 7fd95be51640 1 -- 172.21.15.66:0/853283543 wait complete. 2024-03-25T18:07:06.578 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:06.576+0000 7fd95be51640 1 librados: shutdown 2024-03-25T18:07:06.588 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:07:13.394 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:07:13.508 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.507+0000 7f1142c4e640 10 monclient: get_monmap_and_config 2024-03-25T18:07:13.509 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.507+0000 7f1142c4e640 10 monclient: build_initial_monmap 2024-03-25T18:07:13.509 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.507+0000 7f1142c4e640 10 monclient: monmap: 2024-03-25T18:07:13.509 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:07:13.509 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:13.509 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:07:13.508952+0000 2024-03-25T18:07:13.509 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:07:13.508952+0000 2024-03-25T18:07:13.509 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:13.509 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:13.509 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:07:13.509 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:07:13.509 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:07:13.509 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:13.511 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.508+0000 7f1142c4e640 1 Processor -- start 2024-03-25T18:07:13.511 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.508+0000 7f1142c4e640 1 -- start start 2024-03-25T18:07:13.511 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.508+0000 7f1142c4e640 10 monclient: init 2024-03-25T18:07:13.511 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.509+0000 7f1142c4e640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:13.511 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.509+0000 7f1142c4e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:07:13.511 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.509+0000 7f1142c4e640 10 monclient(hunting): picked mon.noname-c con 0x7f113c0705b0 addr v1:172.21.15.193:6789/0 2024-03-25T18:07:13.511 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.509+0000 7f1142c4e640 10 monclient(hunting): picked mon.noname-a con 0x7f113c1402d0 addr v1:172.21.15.66:6789/0 2024-03-25T18:07:13.511 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.509+0000 7f1142c4e640 10 monclient(hunting): picked mon.noname-b con 0x7f113c149ae0 addr v1:172.21.15.66:6790/0 2024-03-25T18:07:13.511 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.509+0000 7f1142c4e640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c0718c0 con 0x7f113c1402d0 2024-03-25T18:07:13.511 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.509+0000 7f1142c4e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c0714c0 con 0x7f113c0705b0 2024-03-25T18:07:13.511 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.509+0000 7f1142c4e640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c072b50 con 0x7f113c149ae0 2024-03-25T18:07:13.511 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.509+0000 7f1142c4e640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:13.512 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.509+0000 7f1142c4e640 10 monclient(hunting): authenticate will time out at 7182.569506s 2024-03-25T18:07:13.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.516+0000 7f11409ea640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f113c0705b0 0x7f113c13d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:39728/0 (socket says 172.21.15.66:39728) 2024-03-25T18:07:13.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.516+0000 7f11409ea640 1 -- 172.21.15.66:0/3178847900 learned_addr learned my addr 172.21.15.66:0/3178847900 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:07:13.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.529+0000 7f113b7fe640 1 -- 172.21.15.66:0/3178847900 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2657849407 0 0) 0x7f113c0714c0 con 0x7f113c0705b0 2024-03-25T18:07:13.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.529+0000 7f113b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:07:13.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.529+0000 7f113b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:13.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.529+0000 7f113b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:13.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.529+0000 7f113b7fe640 10 monclient(hunting): my global_id is 4369 2024-03-25T18:07:13.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.529+0000 7f113b7fe640 1 -- 172.21.15.66:0/3178847900 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f112c003520 con 0x7f113c0705b0 2024-03-25T18:07:13.534 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.533+0000 7f113b7fe640 1 -- 172.21.15.66:0/3178847900 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1947500418 0 0) 0x7f113c072b50 con 0x7f113c149ae0 2024-03-25T18:07:13.534 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.533+0000 7f113b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:07:13.534 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.533+0000 7f113b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:13.534 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.533+0000 7f113b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:13.534 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.533+0000 7f113b7fe640 10 monclient(hunting): my global_id is 4373 2024-03-25T18:07:13.534 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.533+0000 7f113b7fe640 1 -- 172.21.15.66:0/3178847900 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f113c0714c0 con 0x7f113c149ae0 2024-03-25T18:07:13.546 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.545+0000 7f113b7fe640 1 -- 172.21.15.66:0/3178847900 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 345074145 0 0) 0x7f113c0718c0 con 0x7f113c1402d0 2024-03-25T18:07:13.546 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.545+0000 7f113b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:07:13.546 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.545+0000 7f113b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:13.546 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.545+0000 7f113b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:13.546 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.545+0000 7f113b7fe640 10 monclient(hunting): my global_id is 4374 2024-03-25T18:07:13.546 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.545+0000 7f113b7fe640 1 -- 172.21.15.66:0/3178847900 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f113c072b50 con 0x7f113c1402d0 2024-03-25T18:07:13.555 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.553+0000 7f113b7fe640 1 -- 172.21.15.66:0/3178847900 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4083618560 0 0) 0x7f112c003520 con 0x7f113c0705b0 2024-03-25T18:07:13.555 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.553+0000 7f113b7fe640 1 -- 172.21.15.66:0/3178847900 >> v1:172.21.15.66:6790/0 conn(0x7f113c149ae0 legacy=0x7f113c14bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:13.555 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.553+0000 7f113b7fe640 1 -- 172.21.15.66:0/3178847900 >> v1:172.21.15.66:6789/0 conn(0x7f113c1402d0 legacy=0x7f113c140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:13.555 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.553+0000 7f113b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:07:13.555 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.553+0000 7f113b7fe640 1 monclient: found mon.noname-c 2024-03-25T18:07:13.555 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.553+0000 7f113b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:13.555 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.553+0000 7f113b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:07:13.555 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.553+0000 7f113b7fe640 1 -- 172.21.15.66:0/3178847900 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113c13dd10 con 0x7f113c0705b0 2024-03-25T18:07:13.556 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.553+0000 7f113b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:07:13.556 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.553+0000 7f113b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:07:13.556 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.553+0000 7f113b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:13.556 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.553+0000 7f113b7fe640 1 -- 172.21.15.66:0/3178847900 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f1130001690 con 0x7f113c0705b0 2024-03-25T18:07:13.556 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.553+0000 7f1142c4e640 5 monclient: authenticate success, global_id 4369 2024-03-25T18:07:13.556 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.553+0000 7f1142c4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:13.556 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.553+0000 7f113b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:13.556 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.553+0000 7f113b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:07:13.556 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.553+0000 7f113b7fe640 10 monclient: dump: 2024-03-25T18:07:13.556 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:13.556 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:13.556 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:13.556 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:13.556 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:13.556 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:13.556 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:13.557 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:13.557 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:13.557 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:13.557 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.553+0000 7f1142c4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:13.565 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.564+0000 7f113b7fe640 1 -- 172.21.15.66:0/3178847900 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11300017f0 con 0x7f113c0705b0 2024-03-25T18:07:13.565 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.564+0000 7f113b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:13.565 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.564+0000 7f113b7fe640 1 -- 172.21.15.66:0/3178847900 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f1130001c00 con 0x7f113c0705b0 2024-03-25T18:07:13.565 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.564+0000 7f1142c4e640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:13.565 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.564+0000 7f1142c4e640 4 set_mon_vals no callback set 2024-03-25T18:07:13.566 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.564+0000 7f113b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:13.566 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.564+0000 7f113b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:13.566 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.564+0000 7f113b7fe640 10 monclient: dump: 2024-03-25T18:07:13.566 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:13.566 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:13.566 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:13.566 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:13.566 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:13.566 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:13.566 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:13.566 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:13.566 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:13.566 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:13.568 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.565+0000 7f1142c4e640 10 monclient: shutdown 2024-03-25T18:07:13.568 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.565+0000 7f1142c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:13.568 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.565+0000 7f1142c4e640 1 -- 172.21.15.66:0/3178847900 >> v1:172.21.15.193:6789/0 conn(0x7f113c0705b0 legacy=0x7f113c13d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:13.568 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.565+0000 7f1142c4e640 1 -- 172.21.15.66:0/3178847900 shutdown_connections 2024-03-25T18:07:13.568 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.565+0000 7f1142c4e640 1 -- 172.21.15.66:0/3178847900 >> 172.21.15.66:0/3178847900 conn(0x7f113c0b5d10 msgr2=0x7f113c0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:13.568 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.565+0000 7f1142c4e640 1 -- 172.21.15.66:0/3178847900 shutdown_connections 2024-03-25T18:07:13.568 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.565+0000 7f1142c4e640 1 -- 172.21.15.66:0/3178847900 wait complete. 2024-03-25T18:07:13.568 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.565+0000 7f1142c4e640 10 monclient: build_initial_monmap 2024-03-25T18:07:13.568 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.565+0000 7f1142c4e640 10 monclient: monmap: 2024-03-25T18:07:13.568 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:07:13.568 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:13.568 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:07:13.568 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:07:13.568 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:13.569 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:07:13.569 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:07:13.569 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:07:13.569 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:07:13.569 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:13.569 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.566+0000 7f1142c4e640 1 librados: starting msgr at 2024-03-25T18:07:13.569 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.566+0000 7f1142c4e640 1 librados: starting objecter 2024-03-25T18:07:13.569 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.566+0000 7f1142c4e640 1 Processor -- start 2024-03-25T18:07:13.569 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.566+0000 7f1142c4e640 1 -- start start 2024-03-25T18:07:13.569 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.566+0000 7f1142c4e640 1 librados: setting wanted keys 2024-03-25T18:07:13.569 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.566+0000 7f1142c4e640 1 librados: calling monclient init 2024-03-25T18:07:13.569 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.566+0000 7f1142c4e640 10 monclient: init 2024-03-25T18:07:13.570 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.567+0000 7f1142c4e640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:13.570 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.567+0000 7f1142c4e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:07:13.570 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.567+0000 7f1142c4e640 10 monclient(hunting): picked mon.noname-c con 0x7f113c0705b0 addr v1:172.21.15.66:6790/0 2024-03-25T18:07:13.570 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.567+0000 7f1142c4e640 10 monclient(hunting): picked mon.noname-a con 0x7f113c1402d0 addr v1:172.21.15.66:6789/0 2024-03-25T18:07:13.570 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.567+0000 7f1142c4e640 10 monclient(hunting): picked mon.noname-b con 0x7f113c149ae0 addr v1:172.21.15.193:6789/0 2024-03-25T18:07:13.571 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.567+0000 7f1142c4e640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c1d7540 con 0x7f113c1402d0 2024-03-25T18:07:13.571 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.567+0000 7f1142c4e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c142020 con 0x7f113c149ae0 2024-03-25T18:07:13.571 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.567+0000 7f1142c4e640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f113c1421b0 con 0x7f113c0705b0 2024-03-25T18:07:13.571 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.567+0000 7f1142c4e640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:13.571 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.567+0000 7f1142c4e640 10 monclient(hunting): authenticate will time out at 7182.627842s 2024-03-25T18:07:13.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.573+0000 7f11411eb640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f113c149ae0 0x7f113c1e1f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:39786/0 (socket says 172.21.15.66:39786) 2024-03-25T18:07:13.575 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.573+0000 7f11411eb640 1 -- 172.21.15.66:0/2956754753 learned_addr learned my addr 172.21.15.66:0/2956754753 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:07:13.575 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.574+0000 7f11397fa640 10 client.?.objecter ms_handle_connect 0x7f113c149ae0 2024-03-25T18:07:13.575 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.574+0000 7f11397fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:13.583 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.581+0000 7f11397fa640 1 -- 172.21.15.66:0/2956754753 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 311476759 0 0) 0x7f113c142020 con 0x7f113c149ae0 2024-03-25T18:07:13.583 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.581+0000 7f11397fa640 10 monclient(hunting): _negotiate 2024-03-25T18:07:13.583 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.581+0000 7f11397fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:13.583 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.581+0000 7f11397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:13.583 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.581+0000 7f11397fa640 10 monclient(hunting): my global_id is 4372 2024-03-25T18:07:13.583 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.581+0000 7f11397fa640 1 -- 172.21.15.66:0/2956754753 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1120004eb0 con 0x7f113c149ae0 2024-03-25T18:07:13.587 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.585+0000 7f11397fa640 10 client.?.objecter ms_handle_connect 0x7f113c0705b0 2024-03-25T18:07:13.587 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.585+0000 7f11397fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:13.587 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.586+0000 7f11397fa640 10 client.?.objecter ms_handle_connect 0x7f113c1402d0 2024-03-25T18:07:13.587 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.586+0000 7f11397fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:13.599 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.597+0000 7f11397fa640 1 -- 172.21.15.66:0/2956754753 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597042471 0 0) 0x7f113c1d7540 con 0x7f113c1402d0 2024-03-25T18:07:13.599 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.597+0000 7f11397fa640 10 monclient(hunting): _negotiate 2024-03-25T18:07:13.599 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.597+0000 7f11397fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:13.599 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.597+0000 7f11397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:13.599 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.597+0000 7f11397fa640 10 monclient(hunting): my global_id is 4377 2024-03-25T18:07:13.599 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.597+0000 7f11397fa640 1 -- 172.21.15.66:0/2956754753 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f113c142020 con 0x7f113c1402d0 2024-03-25T18:07:13.606 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.605+0000 7f11397fa640 1 -- 172.21.15.66:0/2956754753 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2868770839 0 0) 0x7f1120004eb0 con 0x7f113c149ae0 2024-03-25T18:07:13.607 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.605+0000 7f11397fa640 1 -- 172.21.15.66:0/2956754753 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f113c1d7540 con 0x7f113c149ae0 2024-03-25T18:07:13.607 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.605+0000 7f11397fa640 1 -- 172.21.15.66:0/2956754753 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f1134001c30 con 0x7f113c149ae0 2024-03-25T18:07:13.607 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.605+0000 7f11397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:13.607 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.605+0000 7f11397fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:07:13.607 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.605+0000 7f11397fa640 10 monclient(hunting): dump: 2024-03-25T18:07:13.607 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:13.607 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:13.607 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:13.607 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:13.607 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:13.607 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:13.607 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:13.607 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:13.607 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:13.608 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:13.608 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.605+0000 7f11397fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:13.608 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.606+0000 7f11397fa640 1 -- 172.21.15.66:0/2956754753 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3384897948 0 0) 0x7f113c1421b0 con 0x7f113c0705b0 2024-03-25T18:07:13.608 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.606+0000 7f11397fa640 10 monclient(hunting): _negotiate 2024-03-25T18:07:13.608 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.606+0000 7f11397fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:13.608 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.606+0000 7f11397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:13.608 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.606+0000 7f11397fa640 10 monclient(hunting): my global_id is 4376 2024-03-25T18:07:13.608 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.606+0000 7f11397fa640 1 -- 172.21.15.66:0/2956754753 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1120004eb0 con 0x7f113c0705b0 2024-03-25T18:07:13.610 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.608+0000 7f11397fa640 1 -- 172.21.15.66:0/2956754753 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2396913824 0 0) 0x7f113c142020 con 0x7f113c1402d0 2024-03-25T18:07:13.610 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.608+0000 7f11397fa640 1 -- 172.21.15.66:0/2956754753 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f113c1421b0 con 0x7f113c1402d0 2024-03-25T18:07:13.610 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.608+0000 7f11397fa640 1 -- 172.21.15.66:0/2956754753 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f11280018b0 con 0x7f113c1402d0 2024-03-25T18:07:13.610 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.608+0000 7f11397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:13.610 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.608+0000 7f11397fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:13.610 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.608+0000 7f11397fa640 10 monclient(hunting): dump: 2024-03-25T18:07:13.610 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:13.610 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:13.610 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:13.610 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:13.610 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:13.610 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:13.610 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:13.611 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:13.611 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:13.611 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:13.618 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.616+0000 7f11397fa640 1 -- 172.21.15.66:0/2956754753 <== mon.0 v1:172.21.15.66:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2619587886 0 0) 0x7f113c1421b0 con 0x7f113c1402d0 2024-03-25T18:07:13.618 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.616+0000 7f11397fa640 1 -- 172.21.15.66:0/2956754753 >> v1:172.21.15.66:6790/0 conn(0x7f113c0705b0 legacy=0x7f113c1d7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:13.618 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.616+0000 7f11397fa640 1 -- 172.21.15.66:0/2956754753 >> v1:172.21.15.193:6789/0 conn(0x7f113c149ae0 legacy=0x7f113c1e1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:13.618 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.616+0000 7f11397fa640 10 monclient: _finish_hunting 0 2024-03-25T18:07:13.618 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.616+0000 7f11397fa640 1 monclient: found mon.a 2024-03-25T18:07:13.618 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.616+0000 7f11397fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:13.618 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.616+0000 7f11397fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:07:13.618 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.616+0000 7f11397fa640 1 -- 172.21.15.66:0/2956754753 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113c1d8170 con 0x7f113c1402d0 2024-03-25T18:07:13.618 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.616+0000 7f11397fa640 10 monclient: _finish_auth 0 2024-03-25T18:07:13.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.616+0000 7f11397fa640 10 monclient: _check_auth_tickets 2024-03-25T18:07:13.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.616+0000 7f11397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:13.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.616+0000 7f1142c4e640 5 monclient: authenticate success, global_id 4377 2024-03-25T18:07:13.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.616+0000 7f1142c4e640 10 monclient: _renew_subs 2024-03-25T18:07:13.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.616+0000 7f1142c4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:07:13.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.616+0000 7f1142c4e640 1 -- 172.21.15.66:0/2956754753 --> v1:172.21.15.66:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f113c0aabc0 con 0x7f113c1402d0 2024-03-25T18:07:13.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.616+0000 7f1142c4e640 10 client.4377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:13.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.616+0000 7f1142c4e640 10 monclient: _renew_subs 2024-03-25T18:07:13.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.616+0000 7f1142c4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:07:13.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.616+0000 7f1142c4e640 1 -- 172.21.15.66:0/2956754753 --> v1:172.21.15.66:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f113c0a99f0 con 0x7f113c1402d0 2024-03-25T18:07:13.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.616+0000 7f1142c4e640 1 librados: init done 2024-03-25T18:07:13.620 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.618+0000 7f1142c4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:13.620 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.618+0000 7f1142c4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:13.620 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.618+0000 7f1142c4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:07:13.620 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.618+0000 7f1142c4e640 1 -- 172.21.15.66:0/2956754753 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f113c0abfc0 con 0x7f113c1402d0 2024-03-25T18:07:13.921 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.919+0000 7f11397fa640 1 -- 172.21.15.66:0/2956754753 <== mon.0 v1:172.21.15.66:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1128003610 con 0x7f113c1402d0 2024-03-25T18:07:13.921 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.919+0000 7f11397fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:13.921 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.919+0000 7f11397fa640 1 -- 172.21.15.66:0/2956754753 <== mon.0 v1:172.21.15.66:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f11280039a0 con 0x7f113c1402d0 2024-03-25T18:07:13.921 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.919+0000 7f11397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:13.921 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.919+0000 7f11397fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:13.921 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.919+0000 7f11397fa640 10 monclient: dump: 2024-03-25T18:07:13.921 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:13.921 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:13.921 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:13.921 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:13.922 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:13.922 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:13.922 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:13.922 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:13.922 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:13.922 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:13.922 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.919+0000 7f113affd640 4 set_mon_vals no callback set 2024-03-25T18:07:13.923 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.920+0000 7f11397fa640 1 -- 172.21.15.66:0/2956754753 <== mon.0 v1:172.21.15.66:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f1128003bc0 con 0x7f113c1402d0 2024-03-25T18:07:13.923 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.921+0000 7f11409ea640 10 monclient: get_auth_request con 0x7f112006df10 auth_method 0 2024-03-25T18:07:13.923 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.921+0000 7f11397fa640 10 client.4377.objecter ms_handle_connect 0x7f112006df10 2024-03-25T18:07:13.928 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.927+0000 7f11397fa640 1 -- 172.21.15.66:0/2956754753 <== mon.0 v1:172.21.15.66:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f11280535b0 con 0x7f113c1402d0 2024-03-25T18:07:13.928 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.927+0000 7f11397fa640 10 client.4377.objecter ms_dispatch 0x7f113c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:07:13.928 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.927+0000 7f11397fa640 3 client.4377.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:07:13.928 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.927+0000 7f11397fa640 3 client.4377.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:07:13.928 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.927+0000 7f11397fa640 20 client.4377.objecter dump_active .. 0 homeless 2024-03-25T18:07:13.928 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.927+0000 7f11397fa640 1 -- 172.21.15.66:0/2956754753 <== mon.0 v1:172.21.15.66:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f112804f0c0 con 0x7f113c1402d0 2024-03-25T18:07:13.928 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.927+0000 7f11397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:13.929 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:13.927+0000 7f11397fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:14.178 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:14.177+0000 7f1142c4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:14.178 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:14.177+0000 7f1142c4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:14.178 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:14.177+0000 7f1142c4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:07:14.178 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:14.177+0000 7f1142c4e640 1 -- 172.21.15.66:0/2956754753 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f113c0627b0 con 0x7f113c1402d0 2024-03-25T18:07:14.209 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:14.208+0000 7f11397fa640 1 -- 172.21.15.66:0/2956754753 <== mon.0 v1:172.21.15.66:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f1128050ff0 con 0x7f113c1402d0 2024-03-25T18:07:14.210 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:14.208+0000 7f11397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:14.210 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:14.208+0000 7f11397fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:14.210 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:07:14.210 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:07:14.213 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:14.211+0000 7f1142c4e640 10 librados: watch_flush enter 2024-03-25T18:07:14.213 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:14.211+0000 7f1142c4e640 10 librados: watch_flush exit 2024-03-25T18:07:14.213 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:14.211+0000 7f1142c4e640 20 client.4377.objecter shutdown clearing up homeless session... 2024-03-25T18:07:14.213 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:14.211+0000 7f1142c4e640 10 client.4377.objecter successfully canceled tick 2024-03-25T18:07:14.213 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:14.211+0000 7f1142c4e640 1 -- 172.21.15.66:0/2956754753 >> v1:172.21.15.193:6800/2275676803 conn(0x7f112006df10 legacy=0x7f11200703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:14.213 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:14.211+0000 7f1142c4e640 10 monclient: shutdown 2024-03-25T18:07:14.213 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:14.211+0000 7f1142c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:14.213 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:14.211+0000 7f1142c4e640 1 -- 172.21.15.66:0/2956754753 >> v1:172.21.15.66:6789/0 conn(0x7f113c1402d0 legacy=0x7f113c0aa4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:14.213 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:14.211+0000 7f1142c4e640 1 -- 172.21.15.66:0/2956754753 shutdown_connections 2024-03-25T18:07:14.213 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:14.211+0000 7f1142c4e640 1 -- 172.21.15.66:0/2956754753 >> 172.21.15.66:0/2956754753 conn(0x7f113c0b5d10 msgr2=0x7f113c0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:14.213 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:14.211+0000 7f1142c4e640 1 -- 172.21.15.66:0/2956754753 shutdown_connections 2024-03-25T18:07:14.213 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:14.211+0000 7f1142c4e640 1 -- 172.21.15.66:0/2956754753 wait complete. 2024-03-25T18:07:14.214 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:14.211+0000 7f1142c4e640 1 librados: shutdown 2024-03-25T18:07:14.225 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:07:21.032 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:07:21.151 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.150+0000 7f23dfe7b640 10 monclient: get_monmap_and_config 2024-03-25T18:07:21.151 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.150+0000 7f23dfe7b640 10 monclient: build_initial_monmap 2024-03-25T18:07:21.151 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.150+0000 7f23dfe7b640 10 monclient: monmap: 2024-03-25T18:07:21.151 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:07:21.151 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:21.151 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:07:21.151609+0000 2024-03-25T18:07:21.152 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:07:21.151609+0000 2024-03-25T18:07:21.152 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:21.152 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:21.152 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:07:21.152 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:07:21.152 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:07:21.152 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:21.153 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.150+0000 7f23dfe7b640 1 Processor -- start 2024-03-25T18:07:21.153 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.151+0000 7f23dfe7b640 1 -- start start 2024-03-25T18:07:21.153 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.151+0000 7f23dfe7b640 10 monclient: init 2024-03-25T18:07:21.153 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.151+0000 7f23dfe7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:21.153 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.151+0000 7f23dfe7b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:07:21.153 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.151+0000 7f23dfe7b640 10 monclient(hunting): picked mon.noname-b con 0x7f23d80a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:07:21.153 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.151+0000 7f23dfe7b640 10 monclient(hunting): picked mon.noname-c con 0x7f23d813f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:07:21.153 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.151+0000 7f23dfe7b640 10 monclient(hunting): picked mon.noname-a con 0x7f23d81430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:07:21.153 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.151+0000 7f23dfe7b640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d8142520 con 0x7f23d81430c0 2024-03-25T18:07:21.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.151+0000 7f23dfe7b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d80b28a0 con 0x7f23d813f710 2024-03-25T18:07:21.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.151+0000 7f23dfe7b640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d80b2a30 con 0x7f23d80a98f0 2024-03-25T18:07:21.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.151+0000 7f23dfe7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:21.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.151+0000 7f23dfe7b640 10 monclient(hunting): authenticate will time out at 7190.211989s 2024-03-25T18:07:21.166 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.165+0000 7f23dd416640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f23d813f710 0x7f23d81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:58480/0 (socket says 172.21.15.66:58480) 2024-03-25T18:07:21.166 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.165+0000 7f23dd416640 1 -- 172.21.15.66:0/1955710221 learned_addr learned my addr 172.21.15.66:0/1955710221 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:07:21.177 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.175+0000 7f23dcc15640 1 -- 172.21.15.66:0/1955710221 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3064396454 0 0) 0x7f23d80b28a0 con 0x7f23d813f710 2024-03-25T18:07:21.177 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.175+0000 7f23dcc15640 10 monclient(hunting): _negotiate 2024-03-25T18:07:21.177 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.175+0000 7f23dcc15640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:21.177 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.175+0000 7f23dcc15640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:21.177 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.175+0000 7f23dcc15640 10 monclient(hunting): my global_id is 4375 2024-03-25T18:07:21.177 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.175+0000 7f23dcc15640 1 -- 172.21.15.66:0/1955710221 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23c4003520 con 0x7f23d813f710 2024-03-25T18:07:21.178 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.177+0000 7f23dcc15640 1 -- 172.21.15.66:0/1955710221 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1948205254 0 0) 0x7f23d80b2a30 con 0x7f23d80a98f0 2024-03-25T18:07:21.179 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.177+0000 7f23dcc15640 10 monclient(hunting): _negotiate 2024-03-25T18:07:21.179 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.177+0000 7f23dcc15640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:21.179 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.177+0000 7f23dcc15640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:21.179 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.177+0000 7f23dcc15640 10 monclient(hunting): my global_id is 4379 2024-03-25T18:07:21.179 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.177+0000 7f23dcc15640 1 -- 172.21.15.66:0/1955710221 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23d80b28a0 con 0x7f23d80a98f0 2024-03-25T18:07:21.185 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.184+0000 7f23dcc15640 1 -- 172.21.15.66:0/1955710221 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 978189071 0 0) 0x7f23d8142520 con 0x7f23d81430c0 2024-03-25T18:07:21.185 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.184+0000 7f23dcc15640 10 monclient(hunting): _negotiate 2024-03-25T18:07:21.185 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.184+0000 7f23dcc15640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:21.185 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.184+0000 7f23dcc15640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:21.185 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.184+0000 7f23dcc15640 10 monclient(hunting): my global_id is 4380 2024-03-25T18:07:21.186 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.184+0000 7f23dcc15640 1 -- 172.21.15.66:0/1955710221 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23d80b2a30 con 0x7f23d81430c0 2024-03-25T18:07:21.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.189+0000 7f23dcc15640 1 -- 172.21.15.66:0/1955710221 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 687274922 0 0) 0x7f23c4003520 con 0x7f23d813f710 2024-03-25T18:07:21.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.189+0000 7f23dcc15640 1 -- 172.21.15.66:0/1955710221 >> v1:172.21.15.66:6790/0 conn(0x7f23d80a98f0 legacy=0x7f23d80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:21.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.189+0000 7f23dcc15640 1 -- 172.21.15.66:0/1955710221 >> v1:172.21.15.66:6789/0 conn(0x7f23d81430c0 legacy=0x7f23d814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:21.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.189+0000 7f23dcc15640 10 monclient: _finish_hunting 0 2024-03-25T18:07:21.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.189+0000 7f23dcc15640 1 monclient: found mon.noname-c 2024-03-25T18:07:21.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.189+0000 7f23dcc15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:21.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.189+0000 7f23dcc15640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:07:21.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.189+0000 7f23dcc15640 1 -- 172.21.15.66:0/1955710221 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d80b44c0 con 0x7f23d813f710 2024-03-25T18:07:21.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.189+0000 7f23dcc15640 10 monclient: _finish_auth 0 2024-03-25T18:07:21.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.189+0000 7f23dcc15640 10 monclient: _check_auth_tickets 2024-03-25T18:07:21.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.189+0000 7f23dcc15640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:21.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.189+0000 7f23dcc15640 1 -- 172.21.15.66:0/1955710221 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f23c8001690 con 0x7f23d813f710 2024-03-25T18:07:21.192 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.189+0000 7f23dcc15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:21.192 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.189+0000 7f23dcc15640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:07:21.192 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.189+0000 7f23dcc15640 10 monclient: dump: 2024-03-25T18:07:21.192 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:21.192 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:21.192 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:21.192 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:21.192 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:21.192 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:21.192 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:21.192 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:21.192 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:21.192 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:21.192 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.189+0000 7f23dfe7b640 5 monclient: authenticate success, global_id 4375 2024-03-25T18:07:21.192 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.189+0000 7f23dfe7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:21.197 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.195+0000 7f23dcc15640 1 -- 172.21.15.66:0/1955710221 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23c80017f0 con 0x7f23d813f710 2024-03-25T18:07:21.197 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.195+0000 7f23dcc15640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:21.197 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.195+0000 7f23dcc15640 1 -- 172.21.15.66:0/1955710221 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f23c8001c00 con 0x7f23d813f710 2024-03-25T18:07:21.197 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.195+0000 7f23dcc15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:21.197 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.195+0000 7f23dcc15640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:21.197 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.195+0000 7f23dcc15640 10 monclient: dump: 2024-03-25T18:07:21.197 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:21.197 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:21.197 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:21.197 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:21.197 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:21.198 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:21.198 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:21.198 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:21.198 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:21.198 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:21.198 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.195+0000 7f23dfe7b640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:21.198 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.195+0000 7f23dfe7b640 4 set_mon_vals no callback set 2024-03-25T18:07:21.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.196+0000 7f23dfe7b640 10 monclient: shutdown 2024-03-25T18:07:21.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.196+0000 7f23dfe7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:21.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.196+0000 7f23dfe7b640 1 -- 172.21.15.66:0/1955710221 >> v1:172.21.15.193:6789/0 conn(0x7f23d813f710 legacy=0x7f23d81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:21.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.196+0000 7f23dfe7b640 1 -- 172.21.15.66:0/1955710221 shutdown_connections 2024-03-25T18:07:21.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.196+0000 7f23dfe7b640 1 -- 172.21.15.66:0/1955710221 >> 172.21.15.66:0/1955710221 conn(0x7f23d80b7ea0 msgr2=0x7f23d80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:21.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.197+0000 7f23dfe7b640 1 -- 172.21.15.66:0/1955710221 shutdown_connections 2024-03-25T18:07:21.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.197+0000 7f23dfe7b640 1 -- 172.21.15.66:0/1955710221 wait complete. 2024-03-25T18:07:21.200 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.197+0000 7f23dfe7b640 10 monclient: build_initial_monmap 2024-03-25T18:07:21.200 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.197+0000 7f23dfe7b640 10 monclient: monmap: 2024-03-25T18:07:21.200 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:07:21.200 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:21.200 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:07:21.200 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:07:21.200 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:21.200 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:07:21.200 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:07:21.200 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:07:21.200 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:07:21.200 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:21.200 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.198+0000 7f23dfe7b640 1 librados: starting msgr at 2024-03-25T18:07:21.200 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.198+0000 7f23dfe7b640 1 librados: starting objecter 2024-03-25T18:07:21.200 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.198+0000 7f23dfe7b640 1 Processor -- start 2024-03-25T18:07:21.200 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.198+0000 7f23dfe7b640 1 -- start start 2024-03-25T18:07:21.201 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.198+0000 7f23dfe7b640 1 librados: setting wanted keys 2024-03-25T18:07:21.201 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.198+0000 7f23dfe7b640 1 librados: calling monclient init 2024-03-25T18:07:21.201 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.198+0000 7f23dfe7b640 10 monclient: init 2024-03-25T18:07:21.201 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.199+0000 7f23dfe7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:21.201 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.199+0000 7f23dfe7b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:07:21.202 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.199+0000 7f23dfe7b640 10 monclient(hunting): picked mon.noname-b con 0x7f23d80a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:07:21.202 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.199+0000 7f23dfe7b640 10 monclient(hunting): picked mon.noname-a con 0x7f23d813f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:07:21.202 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.199+0000 7f23dfe7b640 10 monclient(hunting): picked mon.noname-c con 0x7f23d81430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:07:21.202 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.199+0000 7f23dfe7b640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d81d7490 con 0x7f23d813f710 2024-03-25T18:07:21.202 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.199+0000 7f23dfe7b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d81d76c0 con 0x7f23d80a98f0 2024-03-25T18:07:21.202 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.199+0000 7f23dfe7b640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d81d82f0 con 0x7f23d81430c0 2024-03-25T18:07:21.202 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.199+0000 7f23dfe7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:21.202 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.199+0000 7f23dfe7b640 10 monclient(hunting): authenticate will time out at 7190.259280s 2024-03-25T18:07:21.215 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.214+0000 7f23ddc17640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f23d80a98f0 0x7f23d81d7be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:58518/0 (socket says 172.21.15.66:58518) 2024-03-25T18:07:21.215 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.214+0000 7f23ddc17640 1 -- 172.21.15.66:0/1843350011 learned_addr learned my addr 172.21.15.66:0/1843350011 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:07:21.218 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.217+0000 7f23ce7fc640 10 client.?.objecter ms_handle_connect 0x7f23d81430c0 2024-03-25T18:07:21.218 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.217+0000 7f23ce7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:21.219 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.217+0000 7f23ce7fc640 10 client.?.objecter ms_handle_connect 0x7f23d80a98f0 2024-03-25T18:07:21.219 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.217+0000 7f23ce7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:21.225 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.224+0000 7f23ce7fc640 1 -- 172.21.15.66:0/1843350011 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3663607751 0 0) 0x7f23d81d82f0 con 0x7f23d81430c0 2024-03-25T18:07:21.225 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.224+0000 7f23ce7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:07:21.225 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.224+0000 7f23ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:21.225 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.224+0000 7f23ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:21.225 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.224+0000 7f23ce7fc640 10 monclient(hunting): my global_id is 4382 2024-03-25T18:07:21.225 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.224+0000 7f23ce7fc640 1 -- 172.21.15.66:0/1843350011 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23bc004eb0 con 0x7f23d81430c0 2024-03-25T18:07:21.229 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.227+0000 7f23ce7fc640 1 -- 172.21.15.66:0/1843350011 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1003707577 0 0) 0x7f23d81d76c0 con 0x7f23d80a98f0 2024-03-25T18:07:21.229 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.227+0000 7f23ce7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:07:21.229 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.227+0000 7f23ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:21.229 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.227+0000 7f23ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:21.229 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.227+0000 7f23ce7fc640 10 monclient(hunting): my global_id is 4378 2024-03-25T18:07:21.229 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.228+0000 7f23ce7fc640 1 -- 172.21.15.66:0/1843350011 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23d81d82f0 con 0x7f23d80a98f0 2024-03-25T18:07:21.239 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.238+0000 7f23ce7fc640 1 -- 172.21.15.66:0/1843350011 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2827212431 0 0) 0x7f23d81d82f0 con 0x7f23d80a98f0 2024-03-25T18:07:21.239 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.238+0000 7f23ce7fc640 1 -- 172.21.15.66:0/1843350011 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23d81d76c0 con 0x7f23d80a98f0 2024-03-25T18:07:21.240 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.238+0000 7f23ce7fc640 1 -- 172.21.15.66:0/1843350011 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f23d40019f0 con 0x7f23d80a98f0 2024-03-25T18:07:21.240 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.238+0000 7f23ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:21.240 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.238+0000 7f23ce7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:07:21.240 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.238+0000 7f23ce7fc640 10 monclient(hunting): dump: 2024-03-25T18:07:21.240 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:21.240 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:21.240 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:21.240 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:21.240 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:21.240 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:21.240 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:21.240 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:21.240 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:21.240 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:21.240 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.238+0000 7f23ce7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:21.244 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.243+0000 7f23ce7fc640 10 client.?.objecter ms_handle_connect 0x7f23d813f710 2024-03-25T18:07:21.245 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.243+0000 7f23ce7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:21.248 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.247+0000 7f23ce7fc640 1 -- 172.21.15.66:0/1843350011 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1323618659 0 0) 0x7f23bc004eb0 con 0x7f23d81430c0 2024-03-25T18:07:21.249 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.247+0000 7f23ce7fc640 1 -- 172.21.15.66:0/1843350011 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23d81d82f0 con 0x7f23d81430c0 2024-03-25T18:07:21.249 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.247+0000 7f23ce7fc640 1 -- 172.21.15.66:0/1843350011 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f23d0001bd0 con 0x7f23d81430c0 2024-03-25T18:07:21.249 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.247+0000 7f23ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:21.249 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.247+0000 7f23ce7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:21.249 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.247+0000 7f23ce7fc640 10 monclient(hunting): dump: 2024-03-25T18:07:21.249 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:21.249 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:21.249 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:21.249 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:21.249 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:21.249 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:21.249 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:21.249 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:21.249 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:21.249 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:21.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.252+0000 7f23ce7fc640 1 -- 172.21.15.66:0/1843350011 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3672956945 0 0) 0x7f23d81d76c0 con 0x7f23d80a98f0 2024-03-25T18:07:21.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.252+0000 7f23ce7fc640 1 -- 172.21.15.66:0/1843350011 >> v1:172.21.15.66:6790/0 conn(0x7f23d81430c0 legacy=0x7f23d8133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:21.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.252+0000 7f23ce7fc640 1 -- 172.21.15.66:0/1843350011 >> v1:172.21.15.66:6789/0 conn(0x7f23d813f710 legacy=0x7f23d81383e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:21.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.252+0000 7f23ce7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:07:21.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.252+0000 7f23ce7fc640 1 monclient: found mon.b 2024-03-25T18:07:21.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.252+0000 7f23ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:21.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.252+0000 7f23ce7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:07:21.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.252+0000 7f23ce7fc640 1 -- 172.21.15.66:0/1843350011 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d8138af0 con 0x7f23d80a98f0 2024-03-25T18:07:21.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.252+0000 7f23ce7fc640 10 monclient: _finish_auth 0 2024-03-25T18:07:21.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.252+0000 7f23ce7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:07:21.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.252+0000 7f23ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:21.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.252+0000 7f23dfe7b640 5 monclient: authenticate success, global_id 4378 2024-03-25T18:07:21.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.252+0000 7f23dfe7b640 10 monclient: _renew_subs 2024-03-25T18:07:21.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.252+0000 7f23dfe7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:07:21.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.252+0000 7f23dfe7b640 1 -- 172.21.15.66:0/1843350011 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23d81376d0 con 0x7f23d80a98f0 2024-03-25T18:07:21.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.252+0000 7f23dfe7b640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:21.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.252+0000 7f23dfe7b640 10 monclient: _renew_subs 2024-03-25T18:07:21.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.252+0000 7f23dfe7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:07:21.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.252+0000 7f23dfe7b640 1 -- 172.21.15.66:0/1843350011 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23d8134fc0 con 0x7f23d80a98f0 2024-03-25T18:07:21.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.252+0000 7f23dfe7b640 1 librados: init done 2024-03-25T18:07:21.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.253+0000 7f23dfe7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:21.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.253+0000 7f23dfe7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:21.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.253+0000 7f23dfe7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:07:21.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.253+0000 7f23dfe7b640 1 -- 172.21.15.66:0/1843350011 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23d81d7850 con 0x7f23d80a98f0 2024-03-25T18:07:21.562 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.560+0000 7f23ce7fc640 1 -- 172.21.15.66:0/1843350011 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23d4003610 con 0x7f23d80a98f0 2024-03-25T18:07:21.562 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.560+0000 7f23ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:21.562 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.560+0000 7f23ce7fc640 1 -- 172.21.15.66:0/1843350011 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f23d40039a0 con 0x7f23d80a98f0 2024-03-25T18:07:21.562 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.560+0000 7f23ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:21.562 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.560+0000 7f23ce7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:21.562 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.560+0000 7f23ce7fc640 10 monclient: dump: 2024-03-25T18:07:21.562 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:21.562 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:21.563 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:21.563 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:21.563 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:21.563 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:21.563 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:21.563 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:21.563 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:21.563 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:21.563 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.560+0000 7f23cffff640 4 set_mon_vals no callback set 2024-03-25T18:07:21.563 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.561+0000 7f23ce7fc640 1 -- 172.21.15.66:0/1843350011 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f23d4003b40 con 0x7f23d80a98f0 2024-03-25T18:07:21.564 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.562+0000 7f23dd416640 10 monclient: get_auth_request con 0x7f23bc06de40 auth_method 0 2024-03-25T18:07:21.564 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.562+0000 7f23ce7fc640 10 client.4378.objecter ms_handle_connect 0x7f23bc06de40 2024-03-25T18:07:21.567 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.566+0000 7f23ce7fc640 1 -- 172.21.15.66:0/1843350011 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f23d4054880 con 0x7f23d80a98f0 2024-03-25T18:07:21.567 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.566+0000 7f23ce7fc640 10 client.4378.objecter ms_dispatch 0x7f23d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:07:21.567 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.566+0000 7f23ce7fc640 3 client.4378.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:07:21.567 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.566+0000 7f23ce7fc640 3 client.4378.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:07:21.567 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.566+0000 7f23ce7fc640 20 client.4378.objecter dump_active .. 0 homeless 2024-03-25T18:07:21.568 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.567+0000 7f23ce7fc640 1 -- 172.21.15.66:0/1843350011 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f23d40501c0 con 0x7f23d80a98f0 2024-03-25T18:07:21.568 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.567+0000 7f23ce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:21.568 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.567+0000 7f23ce7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:21.816 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.814+0000 7f23dfe7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:21.816 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.814+0000 7f23dfe7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:21.816 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.814+0000 7f23dfe7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:07:21.816 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.814+0000 7f23dfe7b640 1 -- 172.21.15.66:0/1843350011 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23d814f300 con 0x7f23d80a98f0 2024-03-25T18:07:21.862 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.860+0000 7f23ce7fc640 1 -- 172.21.15.66:0/1843350011 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f23d40522c0 con 0x7f23d80a98f0 2024-03-25T18:07:21.862 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.860+0000 7f23ce7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:21.862 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.860+0000 7f23ce7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:21.862 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:07:21.862 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:07:21.864 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.862+0000 7f23dfe7b640 10 librados: watch_flush enter 2024-03-25T18:07:21.865 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.863+0000 7f23dfe7b640 10 librados: watch_flush exit 2024-03-25T18:07:21.865 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.863+0000 7f23dfe7b640 20 client.4378.objecter shutdown clearing up homeless session... 2024-03-25T18:07:21.865 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.863+0000 7f23dfe7b640 10 client.4378.objecter successfully canceled tick 2024-03-25T18:07:21.865 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.863+0000 7f23dfe7b640 1 -- 172.21.15.66:0/1843350011 >> v1:172.21.15.193:6800/2275676803 conn(0x7f23bc06de40 legacy=0x7f23bc070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:21.865 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.863+0000 7f23dfe7b640 10 monclient: shutdown 2024-03-25T18:07:21.865 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.863+0000 7f23dfe7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:21.865 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.863+0000 7f23dfe7b640 1 -- 172.21.15.66:0/1843350011 >> v1:172.21.15.193:6789/0 conn(0x7f23d80a98f0 legacy=0x7f23d81d7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:21.865 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.863+0000 7f23dfe7b640 1 -- 172.21.15.66:0/1843350011 shutdown_connections 2024-03-25T18:07:21.865 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.863+0000 7f23dfe7b640 1 -- 172.21.15.66:0/1843350011 >> 172.21.15.66:0/1843350011 conn(0x7f23d80b7ea0 msgr2=0x7f23d8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:21.865 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.863+0000 7f23dfe7b640 1 -- 172.21.15.66:0/1843350011 shutdown_connections 2024-03-25T18:07:21.865 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.863+0000 7f23dfe7b640 1 -- 172.21.15.66:0/1843350011 wait complete. 2024-03-25T18:07:21.865 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:21.863+0000 7f23dfe7b640 1 librados: shutdown 2024-03-25T18:07:21.876 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:07:24.129 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:03:55.110 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:24.134 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:03:55.115 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:24.172 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:03:55.153 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:24.189 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:03:55.148 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:24.194 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:03:55.175 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:24.200 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:03:55.159 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:24.202 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:03:55.183 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:24.229 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:03:55.188 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:24.230 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:03:55.210 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:28.682 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:07:28.806 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.804+0000 7ffa609c7640 10 monclient: get_monmap_and_config 2024-03-25T18:07:28.806 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.804+0000 7ffa609c7640 10 monclient: build_initial_monmap 2024-03-25T18:07:28.806 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.804+0000 7ffa609c7640 10 monclient: monmap: 2024-03-25T18:07:28.806 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:07:28.806 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:28.806 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:07:28.806059+0000 2024-03-25T18:07:28.806 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:07:28.806059+0000 2024-03-25T18:07:28.806 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:28.806 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:28.806 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:07:28.807 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:07:28.807 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:07:28.807 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:28.808 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.805+0000 7ffa609c7640 1 Processor -- start 2024-03-25T18:07:28.808 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.805+0000 7ffa609c7640 1 -- start start 2024-03-25T18:07:28.808 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.805+0000 7ffa609c7640 10 monclient: init 2024-03-25T18:07:28.808 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.805+0000 7ffa609c7640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:28.808 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.805+0000 7ffa609c7640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:07:28.808 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.805+0000 7ffa609c7640 10 monclient(hunting): picked mon.noname-c con 0x7ffa5c0a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:07:28.809 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.805+0000 7ffa609c7640 10 monclient(hunting): picked mon.noname-a con 0x7ffa5c13f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:07:28.809 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.805+0000 7ffa609c7640 10 monclient(hunting): picked mon.noname-b con 0x7ffa5c1430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:07:28.809 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.805+0000 7ffa609c7640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa5c142520 con 0x7ffa5c13f710 2024-03-25T18:07:28.809 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.806+0000 7ffa609c7640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa5c0b28a0 con 0x7ffa5c0a98f0 2024-03-25T18:07:28.809 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.806+0000 7ffa609c7640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa5c0b2a30 con 0x7ffa5c1430c0 2024-03-25T18:07:28.809 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.806+0000 7ffa609c7640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:28.809 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.806+0000 7ffa609c7640 10 monclient(hunting): authenticate will time out at 7197.866195s 2024-03-25T18:07:28.821 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.820+0000 7ffa5ad9d640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7ffa5c1430c0 0x7ffa5c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:58334/0 (socket says 172.21.15.66:58334) 2024-03-25T18:07:28.821 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.820+0000 7ffa5ad9d640 1 -- 172.21.15.66:0/923966432 learned_addr learned my addr 172.21.15.66:0/923966432 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:07:28.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.828+0000 7ffa59d9b640 1 -- 172.21.15.66:0/923966432 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3692329119 0 0) 0x7ffa5c0b2a30 con 0x7ffa5c1430c0 2024-03-25T18:07:28.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.828+0000 7ffa59d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:07:28.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.828+0000 7ffa59d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:28.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.828+0000 7ffa59d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:28.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.828+0000 7ffa59d9b640 10 monclient(hunting): my global_id is 4385 2024-03-25T18:07:28.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.828+0000 7ffa59d9b640 1 -- 172.21.15.66:0/923966432 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa48003520 con 0x7ffa5c1430c0 2024-03-25T18:07:28.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.835+0000 7ffa59d9b640 1 -- 172.21.15.66:0/923966432 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2003199874 0 0) 0x7ffa48003520 con 0x7ffa5c1430c0 2024-03-25T18:07:28.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.835+0000 7ffa59d9b640 1 -- 172.21.15.66:0/923966432 >> v1:172.21.15.193:6789/0 conn(0x7ffa5c0a98f0 legacy=0x7ffa5c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:28.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.835+0000 7ffa59d9b640 1 -- 172.21.15.66:0/923966432 >> v1:172.21.15.66:6789/0 conn(0x7ffa5c13f710 legacy=0x7ffa5c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:07:28.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.835+0000 7ffa59d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:07:28.838 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.835+0000 7ffa59d9b640 1 monclient: found mon.noname-b 2024-03-25T18:07:28.838 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.835+0000 7ffa59d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:28.838 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.835+0000 7ffa59d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.66:6790/0 2024-03-25T18:07:28.838 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.835+0000 7ffa59d9b640 1 -- 172.21.15.66:0/923966432 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa5c0b44c0 con 0x7ffa5c1430c0 2024-03-25T18:07:28.838 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.835+0000 7ffa59d9b640 10 monclient: _finish_auth 0 2024-03-25T18:07:28.838 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.835+0000 7ffa59d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:07:28.838 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.835+0000 7ffa59d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:28.838 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.836+0000 7ffa59d9b640 1 -- 172.21.15.66:0/923966432 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7ffa54001a10 con 0x7ffa5c1430c0 2024-03-25T18:07:28.838 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.836+0000 7ffa59d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:28.838 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.836+0000 7ffa59d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:07:28.838 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.836+0000 7ffa59d9b640 10 monclient: dump: 2024-03-25T18:07:28.838 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:28.838 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:28.838 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:28.839 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:28.839 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:28.839 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:28.839 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:28.839 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:28.839 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:28.839 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:28.839 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.836+0000 7ffa609c7640 5 monclient: authenticate success, global_id 4385 2024-03-25T18:07:28.839 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.836+0000 7ffa609c7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:28.851 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.849+0000 7ffa59d9b640 1 -- 172.21.15.66:0/923966432 <== mon.2 v1:172.21.15.66:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa54001be0 con 0x7ffa5c1430c0 2024-03-25T18:07:28.851 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.849+0000 7ffa59d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:28.851 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.849+0000 7ffa59d9b640 1 -- 172.21.15.66:0/923966432 <== mon.2 v1:172.21.15.66:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7ffa54001ff0 con 0x7ffa5c1430c0 2024-03-25T18:07:28.851 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.849+0000 7ffa609c7640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:28.851 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.849+0000 7ffa609c7640 4 set_mon_vals no callback set 2024-03-25T18:07:28.851 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.849+0000 7ffa59d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:28.851 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.849+0000 7ffa59d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:28.851 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.849+0000 7ffa59d9b640 10 monclient: dump: 2024-03-25T18:07:28.851 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:28.851 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:28.851 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:28.851 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:28.851 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:28.852 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:28.852 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:28.852 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:28.852 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:28.852 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:28.853 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.850+0000 7ffa609c7640 10 monclient: shutdown 2024-03-25T18:07:28.853 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.850+0000 7ffa609c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:28.853 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.850+0000 7ffa609c7640 1 -- 172.21.15.66:0/923966432 >> v1:172.21.15.66:6790/0 conn(0x7ffa5c1430c0 legacy=0x7ffa5c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:28.854 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.850+0000 7ffa609c7640 1 -- 172.21.15.66:0/923966432 shutdown_connections 2024-03-25T18:07:28.854 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.850+0000 7ffa609c7640 1 -- 172.21.15.66:0/923966432 >> 172.21.15.66:0/923966432 conn(0x7ffa5c0b7ea0 msgr2=0x7ffa5c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:28.854 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.850+0000 7ffa609c7640 1 -- 172.21.15.66:0/923966432 shutdown_connections 2024-03-25T18:07:28.854 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.850+0000 7ffa609c7640 1 -- 172.21.15.66:0/923966432 wait complete. 2024-03-25T18:07:28.854 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.851+0000 7ffa609c7640 10 monclient: build_initial_monmap 2024-03-25T18:07:28.854 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.851+0000 7ffa609c7640 10 monclient: monmap: 2024-03-25T18:07:28.854 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:07:28.854 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:28.854 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:07:28.854 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:07:28.854 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:28.854 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:07:28.854 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:07:28.854 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:07:28.855 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:07:28.855 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:28.855 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.851+0000 7ffa609c7640 1 librados: starting msgr at 2024-03-25T18:07:28.855 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.851+0000 7ffa609c7640 1 librados: starting objecter 2024-03-25T18:07:28.855 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.852+0000 7ffa609c7640 1 Processor -- start 2024-03-25T18:07:28.855 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.852+0000 7ffa609c7640 1 -- start start 2024-03-25T18:07:28.855 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.852+0000 7ffa609c7640 1 librados: setting wanted keys 2024-03-25T18:07:28.855 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.852+0000 7ffa609c7640 1 librados: calling monclient init 2024-03-25T18:07:28.855 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.852+0000 7ffa609c7640 10 monclient: init 2024-03-25T18:07:28.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.852+0000 7ffa609c7640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:28.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.852+0000 7ffa609c7640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:07:28.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.852+0000 7ffa609c7640 10 monclient(hunting): picked mon.noname-c con 0x7ffa5c0a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:07:28.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.853+0000 7ffa609c7640 10 monclient(hunting): picked mon.noname-b con 0x7ffa5c13f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:07:28.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.853+0000 7ffa609c7640 10 monclient(hunting): picked mon.noname-a con 0x7ffa5c1430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:07:28.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.853+0000 7ffa609c7640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa5c1d7830 con 0x7ffa5c1430c0 2024-03-25T18:07:28.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.853+0000 7ffa609c7640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa5c0ab9e0 con 0x7ffa5c13f710 2024-03-25T18:07:28.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.853+0000 7ffa609c7640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa5c1d7500 con 0x7ffa5c0a98f0 2024-03-25T18:07:28.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.853+0000 7ffa609c7640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:28.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.853+0000 7ffa609c7640 10 monclient(hunting): authenticate will time out at 7197.913247s 2024-03-25T18:07:28.861 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.860+0000 7ffa51d9b640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7ffa5c13f710 0x7ffa5c1d7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:46176/0 (socket says 172.21.15.66:46176) 2024-03-25T18:07:28.861 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.860+0000 7ffa51d9b640 1 -- 172.21.15.66:0/2083662050 learned_addr learned my addr 172.21.15.66:0/2083662050 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:07:28.864 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.863+0000 7ffa537fe640 10 client.?.objecter ms_handle_connect 0x7ffa5c13f710 2024-03-25T18:07:28.864 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.863+0000 7ffa537fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:28.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.872+0000 7ffa537fe640 10 client.?.objecter ms_handle_connect 0x7ffa5c0a98f0 2024-03-25T18:07:28.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.872+0000 7ffa537fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:28.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.875+0000 7ffa537fe640 10 client.?.objecter ms_handle_connect 0x7ffa5c1430c0 2024-03-25T18:07:28.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.875+0000 7ffa537fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:28.880 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.879+0000 7ffa537fe640 1 -- 172.21.15.66:0/2083662050 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483913382 0 0) 0x7ffa5c0ab9e0 con 0x7ffa5c13f710 2024-03-25T18:07:28.880 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.879+0000 7ffa537fe640 10 monclient(hunting): _negotiate 2024-03-25T18:07:28.880 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.879+0000 7ffa537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:28.880 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.879+0000 7ffa537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:28.880 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.879+0000 7ffa537fe640 10 monclient(hunting): my global_id is 4381 2024-03-25T18:07:28.880 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.879+0000 7ffa537fe640 1 -- 172.21.15.66:0/2083662050 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa3c004eb0 con 0x7ffa5c13f710 2024-03-25T18:07:28.889 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.888+0000 7ffa537fe640 1 -- 172.21.15.66:0/2083662050 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1971358721 0 0) 0x7ffa5c1d7500 con 0x7ffa5c0a98f0 2024-03-25T18:07:28.889 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.888+0000 7ffa537fe640 10 monclient(hunting): _negotiate 2024-03-25T18:07:28.889 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.888+0000 7ffa537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:28.889 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.888+0000 7ffa537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:28.889 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.888+0000 7ffa537fe640 10 monclient(hunting): my global_id is 4388 2024-03-25T18:07:28.889 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.888+0000 7ffa537fe640 1 -- 172.21.15.66:0/2083662050 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa5c0ab9e0 con 0x7ffa5c0a98f0 2024-03-25T18:07:28.904 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.902+0000 7ffa537fe640 1 -- 172.21.15.66:0/2083662050 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 375547980 0 0) 0x7ffa5c1d7830 con 0x7ffa5c1430c0 2024-03-25T18:07:28.904 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.902+0000 7ffa537fe640 10 monclient(hunting): _negotiate 2024-03-25T18:07:28.904 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.902+0000 7ffa537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:28.904 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.902+0000 7ffa537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:28.904 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.902+0000 7ffa537fe640 10 monclient(hunting): my global_id is 4386 2024-03-25T18:07:28.904 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.902+0000 7ffa537fe640 1 -- 172.21.15.66:0/2083662050 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa5c1d7500 con 0x7ffa5c1430c0 2024-03-25T18:07:28.910 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.909+0000 7ffa537fe640 1 -- 172.21.15.66:0/2083662050 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1187193514 0 0) 0x7ffa3c004eb0 con 0x7ffa5c13f710 2024-03-25T18:07:28.910 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.909+0000 7ffa537fe640 1 -- 172.21.15.66:0/2083662050 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa5c1d7830 con 0x7ffa5c13f710 2024-03-25T18:07:28.910 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.909+0000 7ffa537fe640 1 -- 172.21.15.66:0/2083662050 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7ffa44004080 con 0x7ffa5c13f710 2024-03-25T18:07:28.910 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.909+0000 7ffa537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:28.911 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.909+0000 7ffa537fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:07:28.911 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.909+0000 7ffa537fe640 10 monclient(hunting): dump: 2024-03-25T18:07:28.911 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:28.911 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:28.911 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:28.911 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:28.911 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:28.911 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:28.911 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:28.911 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:28.911 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:28.911 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:28.911 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.909+0000 7ffa537fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:28.917 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.916+0000 7ffa537fe640 1 -- 172.21.15.66:0/2083662050 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 237181543 0 0) 0x7ffa5c0ab9e0 con 0x7ffa5c0a98f0 2024-03-25T18:07:28.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.916+0000 7ffa537fe640 1 -- 172.21.15.66:0/2083662050 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa3c004eb0 con 0x7ffa5c0a98f0 2024-03-25T18:07:28.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.916+0000 7ffa537fe640 1 -- 172.21.15.66:0/2083662050 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7ffa4c0018b0 con 0x7ffa5c0a98f0 2024-03-25T18:07:28.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.916+0000 7ffa537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:28.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.916+0000 7ffa537fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:28.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.916+0000 7ffa537fe640 10 monclient(hunting): dump: 2024-03-25T18:07:28.918 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:28.918 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:28.918 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:28.918 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:28.918 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:28.918 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:28.918 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:28.918 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:28.918 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:28.919 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:28.925 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.924+0000 7ffa537fe640 1 -- 172.21.15.66:0/2083662050 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 605293917 0 0) 0x7ffa5c1d7500 con 0x7ffa5c1430c0 2024-03-25T18:07:28.925 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.924+0000 7ffa537fe640 1 -- 172.21.15.66:0/2083662050 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa5c0ab9e0 con 0x7ffa5c1430c0 2024-03-25T18:07:28.925 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.924+0000 7ffa537fe640 1 -- 172.21.15.66:0/2083662050 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7ffa54001d50 con 0x7ffa5c1430c0 2024-03-25T18:07:28.925 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.924+0000 7ffa537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:28.925 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.924+0000 7ffa537fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:28.925 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.924+0000 7ffa537fe640 10 monclient(hunting): dump: 2024-03-25T18:07:28.925 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:28.925 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:28.925 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:28.926 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:28.926 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:28.926 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:28.926 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:28.926 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:28.926 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:28.926 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:28.928 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.926+0000 7ffa537fe640 1 -- 172.21.15.66:0/2083662050 <== mon.2 v1:172.21.15.66:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1544691667 0 0) 0x7ffa3c004eb0 con 0x7ffa5c0a98f0 2024-03-25T18:07:28.928 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.926+0000 7ffa537fe640 1 -- 172.21.15.66:0/2083662050 >> v1:172.21.15.193:6789/0 conn(0x7ffa5c13f710 legacy=0x7ffa5c1d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:28.928 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.927+0000 7ffa537fe640 1 -- 172.21.15.66:0/2083662050 >> v1:172.21.15.66:6789/0 conn(0x7ffa5c1430c0 legacy=0x7ffa5c1e1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:28.929 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.927+0000 7ffa537fe640 10 monclient: _finish_hunting 0 2024-03-25T18:07:28.929 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.927+0000 7ffa537fe640 1 monclient: found mon.c 2024-03-25T18:07:28.929 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.927+0000 7ffa537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:28.929 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.927+0000 7ffa537fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:07:28.929 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.927+0000 7ffa537fe640 1 -- 172.21.15.66:0/2083662050 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa5c1d83d0 con 0x7ffa5c0a98f0 2024-03-25T18:07:28.929 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.927+0000 7ffa537fe640 10 monclient: _finish_auth 0 2024-03-25T18:07:28.929 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.927+0000 7ffa537fe640 10 monclient: _check_auth_tickets 2024-03-25T18:07:28.929 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.927+0000 7ffa537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:28.929 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.927+0000 7ffa609c7640 5 monclient: authenticate success, global_id 4388 2024-03-25T18:07:28.929 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.927+0000 7ffa609c7640 10 monclient: _renew_subs 2024-03-25T18:07:28.929 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.927+0000 7ffa609c7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:07:28.929 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.927+0000 7ffa609c7640 1 -- 172.21.15.66:0/2083662050 --> v1:172.21.15.66:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa5c1379c0 con 0x7ffa5c0a98f0 2024-03-25T18:07:28.929 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.927+0000 7ffa609c7640 10 client.4388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:28.929 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.927+0000 7ffa609c7640 10 monclient: _renew_subs 2024-03-25T18:07:28.929 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.927+0000 7ffa609c7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:07:28.930 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.927+0000 7ffa609c7640 1 -- 172.21.15.66:0/2083662050 --> v1:172.21.15.66:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa5c1351f0 con 0x7ffa5c0a98f0 2024-03-25T18:07:28.930 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.927+0000 7ffa609c7640 1 librados: init done 2024-03-25T18:07:28.930 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.928+0000 7ffa609c7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:28.930 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.928+0000 7ffa609c7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:28.930 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.928+0000 7ffa609c7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:07:28.930 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:28.928+0000 7ffa609c7640 1 -- 172.21.15.66:0/2083662050 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa5c137cc0 con 0x7ffa5c0a98f0 2024-03-25T18:07:29.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.252+0000 7ffa537fe640 1 -- 172.21.15.66:0/2083662050 <== mon.2 v1:172.21.15.66:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa4c0035f0 con 0x7ffa5c0a98f0 2024-03-25T18:07:29.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.252+0000 7ffa537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:29.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.252+0000 7ffa537fe640 1 -- 172.21.15.66:0/2083662050 <== mon.2 v1:172.21.15.66:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7ffa4c003900 con 0x7ffa5c0a98f0 2024-03-25T18:07:29.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.252+0000 7ffa537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:29.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.252+0000 7ffa537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:29.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.252+0000 7ffa537fe640 10 monclient: dump: 2024-03-25T18:07:29.255 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:29.255 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:29.255 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:29.255 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:29.255 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:29.255 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:29.255 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:29.255 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:29.255 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:29.255 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:29.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.252+0000 7ffa5959a640 4 set_mon_vals no callback set 2024-03-25T18:07:29.256 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.253+0000 7ffa537fe640 1 -- 172.21.15.66:0/2083662050 <== mon.2 v1:172.21.15.66:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7ffa4c0198d0 con 0x7ffa5c0a98f0 2024-03-25T18:07:29.256 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.254+0000 7ffa51d9b640 10 monclient: get_auth_request con 0x7ffa3c06e0c0 auth_method 0 2024-03-25T18:07:29.256 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.254+0000 7ffa537fe640 10 client.4388.objecter ms_handle_connect 0x7ffa3c06e0c0 2024-03-25T18:07:29.260 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.258+0000 7ffa537fe640 1 -- 172.21.15.66:0/2083662050 <== mon.2 v1:172.21.15.66:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7ffa4c053530 con 0x7ffa5c0a98f0 2024-03-25T18:07:29.260 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.258+0000 7ffa537fe640 10 client.4388.objecter ms_dispatch 0x7ffa5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:07:29.260 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.258+0000 7ffa537fe640 3 client.4388.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:07:29.260 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.258+0000 7ffa537fe640 3 client.4388.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:07:29.260 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.258+0000 7ffa537fe640 20 client.4388.objecter dump_active .. 0 homeless 2024-03-25T18:07:29.260 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.258+0000 7ffa537fe640 1 -- 172.21.15.66:0/2083662050 <== mon.2 v1:172.21.15.66:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffa4c04f040 con 0x7ffa5c0a98f0 2024-03-25T18:07:29.260 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.258+0000 7ffa537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:29.260 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.258+0000 7ffa537fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:29.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.519+0000 7ffa609c7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:29.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.519+0000 7ffa609c7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:29.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.519+0000 7ffa609c7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:07:29.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.519+0000 7ffa609c7640 1 -- 172.21.15.66:0/2083662050 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa5c0627b0 con 0x7ffa5c0a98f0 2024-03-25T18:07:29.571 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.569+0000 7ffa537fe640 1 -- 172.21.15.66:0/2083662050 <== mon.2 v1:172.21.15.66:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7ffa4c050f70 con 0x7ffa5c0a98f0 2024-03-25T18:07:29.571 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.569+0000 7ffa537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:29.571 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.569+0000 7ffa537fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:29.571 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:07:29.571 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:07:29.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.572+0000 7ffa609c7640 10 librados: watch_flush enter 2024-03-25T18:07:29.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.572+0000 7ffa609c7640 10 librados: watch_flush exit 2024-03-25T18:07:29.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.572+0000 7ffa609c7640 20 client.4388.objecter shutdown clearing up homeless session... 2024-03-25T18:07:29.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.572+0000 7ffa609c7640 10 client.4388.objecter successfully canceled tick 2024-03-25T18:07:29.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.572+0000 7ffa609c7640 1 -- 172.21.15.66:0/2083662050 >> v1:172.21.15.193:6800/2275676803 conn(0x7ffa3c06e0c0 legacy=0x7ffa3c070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:29.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.572+0000 7ffa609c7640 10 monclient: shutdown 2024-03-25T18:07:29.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.572+0000 7ffa609c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:29.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.572+0000 7ffa609c7640 1 -- 172.21.15.66:0/2083662050 >> v1:172.21.15.66:6790/0 conn(0x7ffa5c0a98f0 legacy=0x7ffa5c138760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:29.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.572+0000 7ffa609c7640 1 -- 172.21.15.66:0/2083662050 shutdown_connections 2024-03-25T18:07:29.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.572+0000 7ffa609c7640 1 -- 172.21.15.66:0/2083662050 >> 172.21.15.66:0/2083662050 conn(0x7ffa5c0b7ea0 msgr2=0x7ffa5c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:29.575 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.572+0000 7ffa609c7640 1 -- 172.21.15.66:0/2083662050 shutdown_connections 2024-03-25T18:07:29.575 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.572+0000 7ffa609c7640 1 -- 172.21.15.66:0/2083662050 wait complete. 2024-03-25T18:07:29.575 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:29.572+0000 7ffa609c7640 1 librados: shutdown 2024-03-25T18:07:29.586 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:07:36.393 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:07:36.495 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.495+0000 7f72938ba640 10 monclient: get_monmap_and_config 2024-03-25T18:07:36.495 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.495+0000 7f72938ba640 10 monclient: build_initial_monmap 2024-03-25T18:07:36.495 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.495+0000 7f72938ba640 10 monclient: monmap: 2024-03-25T18:07:36.495 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:07:36.495 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:36.496 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:07:36.495556+0000 2024-03-25T18:07:36.496 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:07:36.495556+0000 2024-03-25T18:07:36.496 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:36.496 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:36.496 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:07:36.496 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:07:36.496 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:07:36.496 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:36.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.495+0000 7f72938ba640 1 Processor -- start 2024-03-25T18:07:36.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.495+0000 7f72938ba640 1 -- start start 2024-03-25T18:07:36.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.495+0000 7f72938ba640 10 monclient: init 2024-03-25T18:07:36.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.496+0000 7f72938ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:36.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.496+0000 7f72938ba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:07:36.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.496+0000 7f72938ba640 10 monclient(hunting): picked mon.noname-a con 0x7f728c06ab80 addr v1:172.21.15.66:6789/0 2024-03-25T18:07:36.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.496+0000 7f72938ba640 10 monclient(hunting): picked mon.noname-c con 0x7f728c06af60 addr v1:172.21.15.193:6789/0 2024-03-25T18:07:36.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.496+0000 7f72938ba640 10 monclient(hunting): picked mon.noname-b con 0x7f728c0b00b0 addr v1:172.21.15.66:6790/0 2024-03-25T18:07:36.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.496+0000 7f72938ba640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f728c0b0500 con 0x7f728c06ab80 2024-03-25T18:07:36.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.496+0000 7f72938ba640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f728c13ffa0 con 0x7f728c06af60 2024-03-25T18:07:36.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.496+0000 7f72938ba640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f728c0af520 con 0x7f728c0b00b0 2024-03-25T18:07:36.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.496+0000 7f72938ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:36.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.496+0000 7f72938ba640 10 monclient(hunting): authenticate will time out at 7205.555680s 2024-03-25T18:07:36.510 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.510+0000 7f7291656640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f728c06ab80 0x7f728c0ad880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:51534/0 (socket says 172.21.15.66:51534) 2024-03-25T18:07:36.510 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.510+0000 7f7291656640 1 -- 172.21.15.66:0/2858660793 learned_addr learned my addr 172.21.15.66:0/2858660793 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:07:36.525 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.525+0000 7f7283fff640 1 -- 172.21.15.66:0/2858660793 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 197171118 0 0) 0x7f728c0b0500 con 0x7f728c06ab80 2024-03-25T18:07:36.525 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.525+0000 7f7283fff640 10 monclient(hunting): _negotiate 2024-03-25T18:07:36.526 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.525+0000 7f7283fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:36.526 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.525+0000 7f7283fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:36.526 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.525+0000 7f7283fff640 10 monclient(hunting): my global_id is 4389 2024-03-25T18:07:36.526 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.525+0000 7f7283fff640 1 -- 172.21.15.66:0/2858660793 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7278003520 con 0x7f728c06ab80 2024-03-25T18:07:36.542 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.542+0000 7f7283fff640 1 -- 172.21.15.66:0/2858660793 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1689869096 0 0) 0x7f7278003520 con 0x7f728c06ab80 2024-03-25T18:07:36.542 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.542+0000 7f7283fff640 1 -- 172.21.15.66:0/2858660793 >> v1:172.21.15.66:6790/0 conn(0x7f728c0b00b0 legacy=0x7f728c14dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:07:36.542 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.542+0000 7f7283fff640 1 -- 172.21.15.66:0/2858660793 >> v1:172.21.15.193:6789/0 conn(0x7f728c06af60 legacy=0x7f728c13f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:36.542 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.542+0000 7f7283fff640 10 monclient: _finish_hunting 0 2024-03-25T18:07:36.543 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.542+0000 7f7283fff640 1 monclient: found mon.noname-a 2024-03-25T18:07:36.543 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.542+0000 7f7283fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:36.543 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.542+0000 7f7283fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.66:6789/0 2024-03-25T18:07:36.543 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.542+0000 7f7283fff640 1 -- 172.21.15.66:0/2858660793 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f728c072890 con 0x7f728c06ab80 2024-03-25T18:07:36.543 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.542+0000 7f7283fff640 10 monclient: _finish_auth 0 2024-03-25T18:07:36.543 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.542+0000 7f7283fff640 10 monclient: _check_auth_tickets 2024-03-25T18:07:36.543 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.542+0000 7f7283fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:36.543 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.542+0000 7f7283fff640 1 -- 172.21.15.66:0/2858660793 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7288001690 con 0x7f728c06ab80 2024-03-25T18:07:36.543 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.542+0000 7f7283fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:36.543 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.542+0000 7f7283fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:07:36.543 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.542+0000 7f7283fff640 10 monclient: dump: 2024-03-25T18:07:36.543 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:36.543 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:36.543 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:36.544 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:36.544 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:36.544 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:36.544 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:36.544 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:36.544 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:36.544 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:36.544 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.542+0000 7f72938ba640 5 monclient: authenticate success, global_id 4389 2024-03-25T18:07:36.544 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.542+0000 7f72938ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:36.560 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.560+0000 7f7283fff640 1 -- 172.21.15.66:0/2858660793 <== mon.0 v1:172.21.15.66:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7288001860 con 0x7f728c06ab80 2024-03-25T18:07:36.561 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.560+0000 7f7283fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:36.561 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.560+0000 7f7283fff640 1 -- 172.21.15.66:0/2858660793 <== mon.0 v1:172.21.15.66:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7288001cb0 con 0x7f728c06ab80 2024-03-25T18:07:36.561 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.560+0000 7f72938ba640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:36.561 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.560+0000 7f72938ba640 4 set_mon_vals no callback set 2024-03-25T18:07:36.561 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.560+0000 7f7283fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:36.561 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.560+0000 7f7283fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:36.561 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.560+0000 7f7283fff640 10 monclient: dump: 2024-03-25T18:07:36.561 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:36.561 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:36.561 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:36.562 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:36.562 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:36.562 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:36.562 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:36.562 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:36.562 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:36.562 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:36.564 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.560+0000 7f72938ba640 10 monclient: shutdown 2024-03-25T18:07:36.564 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.560+0000 7f72938ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:36.564 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.560+0000 7f72938ba640 1 -- 172.21.15.66:0/2858660793 >> v1:172.21.15.66:6789/0 conn(0x7f728c06ab80 legacy=0x7f728c0ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:36.564 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.561+0000 7f72938ba640 1 -- 172.21.15.66:0/2858660793 shutdown_connections 2024-03-25T18:07:36.564 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.561+0000 7f72938ba640 1 -- 172.21.15.66:0/2858660793 >> 172.21.15.66:0/2858660793 conn(0x7f728c0b4a80 msgr2=0x7f728c0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:36.564 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.561+0000 7f72938ba640 1 -- 172.21.15.66:0/2858660793 shutdown_connections 2024-03-25T18:07:36.564 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.561+0000 7f72938ba640 1 -- 172.21.15.66:0/2858660793 wait complete. 2024-03-25T18:07:36.565 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.561+0000 7f72938ba640 10 monclient: build_initial_monmap 2024-03-25T18:07:36.565 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.561+0000 7f72938ba640 10 monclient: monmap: 2024-03-25T18:07:36.565 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:07:36.565 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:36.565 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:07:36.565 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:07:36.565 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:36.565 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:07:36.565 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:07:36.565 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:07:36.565 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:07:36.566 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:36.566 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.562+0000 7f72938ba640 1 librados: starting msgr at 2024-03-25T18:07:36.566 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.562+0000 7f72938ba640 1 librados: starting objecter 2024-03-25T18:07:36.566 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.562+0000 7f72938ba640 1 Processor -- start 2024-03-25T18:07:36.566 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.562+0000 7f72938ba640 1 -- start start 2024-03-25T18:07:36.566 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.562+0000 7f72938ba640 1 librados: setting wanted keys 2024-03-25T18:07:36.566 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.562+0000 7f72938ba640 1 librados: calling monclient init 2024-03-25T18:07:36.566 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.562+0000 7f72938ba640 10 monclient: init 2024-03-25T18:07:36.566 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.562+0000 7f72938ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:36.566 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.563+0000 7f72938ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:07:36.566 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.563+0000 7f72938ba640 10 monclient(hunting): picked mon.noname-a con 0x7f728c06ab80 addr v1:172.21.15.66:6789/0 2024-03-25T18:07:36.566 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.563+0000 7f72938ba640 10 monclient(hunting): picked mon.noname-c con 0x7f728c0b00b0 addr v1:172.21.15.66:6790/0 2024-03-25T18:07:36.566 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.563+0000 7f72938ba640 10 monclient(hunting): picked mon.noname-b con 0x7f728c1d4f20 addr v1:172.21.15.193:6789/0 2024-03-25T18:07:36.566 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.563+0000 7f72938ba640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f728c145540 con 0x7f728c06ab80 2024-03-25T18:07:36.567 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.563+0000 7f72938ba640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f728c1467a0 con 0x7f728c1d4f20 2024-03-25T18:07:36.567 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.563+0000 7f72938ba640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f728c144760 con 0x7f728c0b00b0 2024-03-25T18:07:36.567 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.563+0000 7f72938ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:36.567 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.563+0000 7f72938ba640 10 monclient(hunting): authenticate will time out at 7205.622280s 2024-03-25T18:07:36.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.574+0000 7f7290e55640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f728c0b00b0 0x7f728c1d4810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:53804/0 (socket says 172.21.15.66:53804) 2024-03-25T18:07:36.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.574+0000 7f7290e55640 1 -- 172.21.15.66:0/2640150001 learned_addr learned my addr 172.21.15.66:0/2640150001 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:07:36.575 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.575+0000 7f7281ffb640 10 client.?.objecter ms_handle_connect 0x7f728c0b00b0 2024-03-25T18:07:36.575 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.575+0000 7f7281ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:36.577 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.577+0000 7f7281ffb640 10 client.?.objecter ms_handle_connect 0x7f728c06ab80 2024-03-25T18:07:36.577 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.577+0000 7f7281ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:36.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.578+0000 7f7281ffb640 10 client.?.objecter ms_handle_connect 0x7f728c1d4f20 2024-03-25T18:07:36.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.578+0000 7f7281ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:36.586 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.585+0000 7f7281ffb640 1 -- 172.21.15.66:0/2640150001 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2732141849 0 0) 0x7f728c144760 con 0x7f728c0b00b0 2024-03-25T18:07:36.586 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.585+0000 7f7281ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:07:36.586 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.585+0000 7f7281ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:36.586 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.585+0000 7f7281ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:36.586 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.585+0000 7f7281ffb640 10 monclient(hunting): my global_id is 4391 2024-03-25T18:07:36.586 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.585+0000 7f7281ffb640 1 -- 172.21.15.66:0/2640150001 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7270004eb0 con 0x7f728c0b00b0 2024-03-25T18:07:36.591 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.591+0000 7f7281ffb640 1 -- 172.21.15.66:0/2640150001 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2436814024 0 0) 0x7f728c1467a0 con 0x7f728c1d4f20 2024-03-25T18:07:36.592 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.591+0000 7f7281ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:07:36.592 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.591+0000 7f7281ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:36.592 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.591+0000 7f7281ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:36.592 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.591+0000 7f7281ffb640 10 monclient(hunting): my global_id is 4387 2024-03-25T18:07:36.592 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.591+0000 7f7281ffb640 1 -- 172.21.15.66:0/2640150001 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f728c144760 con 0x7f728c1d4f20 2024-03-25T18:07:36.599 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.598+0000 7f7281ffb640 1 -- 172.21.15.66:0/2640150001 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1710788203 0 0) 0x7f728c145540 con 0x7f728c06ab80 2024-03-25T18:07:36.599 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.598+0000 7f7281ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:07:36.599 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.598+0000 7f7281ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:36.599 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.598+0000 7f7281ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:36.599 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.598+0000 7f7281ffb640 10 monclient(hunting): my global_id is 4392 2024-03-25T18:07:36.599 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.598+0000 7f7281ffb640 1 -- 172.21.15.66:0/2640150001 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f728c1467a0 con 0x7f728c06ab80 2024-03-25T18:07:36.601 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.600+0000 7f7281ffb640 1 -- 172.21.15.66:0/2640150001 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 851792733 0 0) 0x7f7270004eb0 con 0x7f728c0b00b0 2024-03-25T18:07:36.601 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.601+0000 7f7281ffb640 1 -- 172.21.15.66:0/2640150001 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f728c145540 con 0x7f728c0b00b0 2024-03-25T18:07:36.601 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.601+0000 7f7281ffb640 1 -- 172.21.15.66:0/2640150001 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f727c001a30 con 0x7f728c0b00b0 2024-03-25T18:07:36.601 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.601+0000 7f7281ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:36.601 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.601+0000 7f7281ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:07:36.601 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.601+0000 7f7281ffb640 10 monclient(hunting): dump: 2024-03-25T18:07:36.601 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:36.602 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:36.602 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:36.602 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:36.602 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:36.602 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:36.602 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:36.602 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:36.602 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:36.602 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:36.602 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.601+0000 7f7281ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:36.611 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.610+0000 7f7281ffb640 1 -- 172.21.15.66:0/2640150001 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 155794121 0 0) 0x7f728c144760 con 0x7f728c1d4f20 2024-03-25T18:07:36.611 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.610+0000 7f7281ffb640 1 -- 172.21.15.66:0/2640150001 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7270004eb0 con 0x7f728c1d4f20 2024-03-25T18:07:36.611 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.610+0000 7f7281ffb640 1 -- 172.21.15.66:0/2640150001 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7284001a70 con 0x7f728c1d4f20 2024-03-25T18:07:36.611 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.610+0000 7f7281ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:36.611 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.610+0000 7f7281ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:36.611 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.610+0000 7f7281ffb640 10 monclient(hunting): dump: 2024-03-25T18:07:36.611 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:36.611 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:36.611 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:36.611 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:36.612 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:36.612 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:36.612 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:36.612 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:36.612 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:36.612 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:36.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.622+0000 7f7281ffb640 1 -- 172.21.15.66:0/2640150001 <== mon.2 v1:172.21.15.66:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3367950142 0 0) 0x7f728c145540 con 0x7f728c0b00b0 2024-03-25T18:07:36.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.622+0000 7f7281ffb640 1 -- 172.21.15.66:0/2640150001 >> v1:172.21.15.193:6789/0 conn(0x7f728c1d4f20 legacy=0x7f728c1e30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:36.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.622+0000 7f7281ffb640 1 -- 172.21.15.66:0/2640150001 >> v1:172.21.15.66:6789/0 conn(0x7f728c06ab80 legacy=0x7f728c146090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:36.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.622+0000 7f7281ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:07:36.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.622+0000 7f7281ffb640 1 monclient: found mon.c 2024-03-25T18:07:36.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.622+0000 7f7281ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:36.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.622+0000 7f7281ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:07:36.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.622+0000 7f7281ffb640 1 -- 172.21.15.66:0/2640150001 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f728c145bb0 con 0x7f728c0b00b0 2024-03-25T18:07:36.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.622+0000 7f7281ffb640 10 monclient: _finish_auth 0 2024-03-25T18:07:36.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.622+0000 7f7281ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:07:36.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.622+0000 7f7281ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:36.624 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.622+0000 7f72938ba640 5 monclient: authenticate success, global_id 4391 2024-03-25T18:07:36.624 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.622+0000 7f72938ba640 10 monclient: _renew_subs 2024-03-25T18:07:36.624 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.622+0000 7f72938ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:07:36.624 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.622+0000 7f72938ba640 1 -- 172.21.15.66:0/2640150001 --> v1:172.21.15.66:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f728c1457f0 con 0x7f728c0b00b0 2024-03-25T18:07:36.624 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.622+0000 7f72938ba640 10 client.4391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:36.624 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.622+0000 7f72938ba640 10 monclient: _renew_subs 2024-03-25T18:07:36.624 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.622+0000 7f72938ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:07:36.624 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.622+0000 7f72938ba640 1 -- 172.21.15.66:0/2640150001 --> v1:172.21.15.66:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f728c1438b0 con 0x7f728c0b00b0 2024-03-25T18:07:36.624 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.622+0000 7f72938ba640 1 librados: init done 2024-03-25T18:07:36.625 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.623+0000 7f72938ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:36.625 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.623+0000 7f72938ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:36.625 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.623+0000 7f72938ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:07:36.625 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.623+0000 7f72938ba640 1 -- 172.21.15.66:0/2640150001 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f728c143280 con 0x7f728c0b00b0 2024-03-25T18:07:36.953 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.952+0000 7f7281ffb640 1 -- 172.21.15.66:0/2640150001 <== mon.2 v1:172.21.15.66:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f727c001c70 con 0x7f728c0b00b0 2024-03-25T18:07:36.953 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.952+0000 7f7281ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:36.953 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.952+0000 7f7281ffb640 1 -- 172.21.15.66:0/2640150001 <== mon.2 v1:172.21.15.66:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f727c003a60 con 0x7f728c0b00b0 2024-03-25T18:07:36.953 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.952+0000 7f7281ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:36.953 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.952+0000 7f7281ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:36.953 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.952+0000 7f7281ffb640 10 monclient: dump: 2024-03-25T18:07:36.953 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:36.953 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:36.953 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:36.953 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:36.953 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:36.953 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:36.953 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:36.954 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:36.954 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:36.954 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:36.954 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.952+0000 7f72837fe640 4 set_mon_vals no callback set 2024-03-25T18:07:36.954 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.953+0000 7f7281ffb640 1 -- 172.21.15.66:0/2640150001 <== mon.2 v1:172.21.15.66:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f727c003c80 con 0x7f728c0b00b0 2024-03-25T18:07:36.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.954+0000 7f7291656640 10 monclient: get_auth_request con 0x7f727006df70 auth_method 0 2024-03-25T18:07:36.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.954+0000 7f7281ffb640 10 client.4391.objecter ms_handle_connect 0x7f727006df70 2024-03-25T18:07:36.959 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.959+0000 7f7281ffb640 1 -- 172.21.15.66:0/2640150001 <== mon.2 v1:172.21.15.66:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f727c054890 con 0x7f728c0b00b0 2024-03-25T18:07:36.959 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.959+0000 7f7281ffb640 10 client.4391.objecter ms_dispatch 0x7f728c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:07:36.959 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.959+0000 7f7281ffb640 3 client.4391.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:07:36.959 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.959+0000 7f7281ffb640 3 client.4391.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:07:36.959 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.959+0000 7f7281ffb640 20 client.4391.objecter dump_active .. 0 homeless 2024-03-25T18:07:36.960 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.959+0000 7f7281ffb640 1 -- 172.21.15.66:0/2640150001 <== mon.2 v1:172.21.15.66:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f727c0503a0 con 0x7f728c0b00b0 2024-03-25T18:07:36.960 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.959+0000 7f7281ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:36.960 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:36.959+0000 7f7281ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:37.217 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:37.217+0000 7f72938ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:37.217 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:37.217+0000 7f72938ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:37.218 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:37.217+0000 7f72938ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:07:37.218 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:37.217+0000 7f72938ba640 1 -- 172.21.15.66:0/2640150001 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f728c0627b0 con 0x7f728c0b00b0 2024-03-25T18:07:37.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:37.253+0000 7f7281ffb640 1 -- 172.21.15.66:0/2640150001 <== mon.2 v1:172.21.15.66:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f727c0522d0 con 0x7f728c0b00b0 2024-03-25T18:07:37.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:37.253+0000 7f7281ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:37.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:37.253+0000 7f7281ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:37.254 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:07:37.254 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:07:37.256 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:37.256+0000 7f72938ba640 10 librados: watch_flush enter 2024-03-25T18:07:37.256 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:37.256+0000 7f72938ba640 10 librados: watch_flush exit 2024-03-25T18:07:37.257 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:37.256+0000 7f72938ba640 20 client.4391.objecter shutdown clearing up homeless session... 2024-03-25T18:07:37.257 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:37.256+0000 7f72938ba640 10 client.4391.objecter successfully canceled tick 2024-03-25T18:07:37.257 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:37.256+0000 7f72938ba640 1 -- 172.21.15.66:0/2640150001 >> v1:172.21.15.193:6800/2275676803 conn(0x7f727006df70 legacy=0x7f7270070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:37.257 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:37.256+0000 7f72938ba640 10 monclient: shutdown 2024-03-25T18:07:37.257 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:37.256+0000 7f72938ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:37.257 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:37.256+0000 7f72938ba640 1 -- 172.21.15.66:0/2640150001 >> v1:172.21.15.66:6790/0 conn(0x7f728c0b00b0 legacy=0x7f728c1d4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:37.257 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:37.256+0000 7f72938ba640 1 -- 172.21.15.66:0/2640150001 shutdown_connections 2024-03-25T18:07:37.257 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:37.256+0000 7f72938ba640 1 -- 172.21.15.66:0/2640150001 >> 172.21.15.66:0/2640150001 conn(0x7f728c0b4a80 msgr2=0x7f728c13f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:37.257 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:37.256+0000 7f72938ba640 1 -- 172.21.15.66:0/2640150001 shutdown_connections 2024-03-25T18:07:37.258 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:37.257+0000 7f72938ba640 1 -- 172.21.15.66:0/2640150001 wait complete. 2024-03-25T18:07:37.258 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:37.257+0000 7f72938ba640 1 librados: shutdown 2024-03-25T18:07:37.268 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:07:44.075 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:07:44.188 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.187+0000 7fea49eae640 10 monclient: get_monmap_and_config 2024-03-25T18:07:44.188 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.187+0000 7fea49eae640 10 monclient: build_initial_monmap 2024-03-25T18:07:44.188 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.187+0000 7fea49eae640 10 monclient: monmap: 2024-03-25T18:07:44.188 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:07:44.188 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:44.188 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:07:44.188437+0000 2024-03-25T18:07:44.188 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:07:44.188437+0000 2024-03-25T18:07:44.188 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:44.189 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:44.189 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:07:44.189 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:07:44.189 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:07:44.189 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:44.190 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.188+0000 7fea49eae640 1 Processor -- start 2024-03-25T18:07:44.190 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.188+0000 7fea49eae640 1 -- start start 2024-03-25T18:07:44.190 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.188+0000 7fea49eae640 10 monclient: init 2024-03-25T18:07:44.190 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.189+0000 7fea49eae640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:44.190 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.189+0000 7fea49eae640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:07:44.190 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.189+0000 7fea49eae640 10 monclient(hunting): picked mon.noname-a con 0x7fea440b4d60 addr v1:172.21.15.66:6789/0 2024-03-25T18:07:44.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.189+0000 7fea49eae640 10 monclient(hunting): picked mon.noname-b con 0x7fea440b1730 addr v1:172.21.15.66:6790/0 2024-03-25T18:07:44.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.189+0000 7fea49eae640 10 monclient(hunting): picked mon.noname-c con 0x7fea4414be90 addr v1:172.21.15.193:6789/0 2024-03-25T18:07:44.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.189+0000 7fea49eae640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea440af940 con 0x7fea440b4d60 2024-03-25T18:07:44.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.189+0000 7fea49eae640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea440b0a70 con 0x7fea4414be90 2024-03-25T18:07:44.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.189+0000 7fea49eae640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea440b0d50 con 0x7fea440b1730 2024-03-25T18:07:44.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.189+0000 7fea49eae640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:44.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.189+0000 7fea49eae640 10 monclient(hunting): authenticate will time out at 7213.248420s 2024-03-25T18:07:44.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.189+0000 7fea42ffd640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7fea440b1730 0x7fea440b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:54054/0 (socket says 172.21.15.66:54054) 2024-03-25T18:07:44.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.189+0000 7fea42ffd640 1 -- 172.21.15.66:0/2301691360 learned_addr learned my addr 172.21.15.66:0/2301691360 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:07:44.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.252+0000 7fea427fc640 1 -- 172.21.15.66:0/2301691360 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1689840363 0 0) 0x7fea440af940 con 0x7fea440b4d60 2024-03-25T18:07:44.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.252+0000 7fea427fc640 10 monclient(hunting): _negotiate 2024-03-25T18:07:44.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.252+0000 7fea427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:44.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.252+0000 7fea427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:44.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.252+0000 7fea427fc640 10 monclient(hunting): my global_id is 4395 2024-03-25T18:07:44.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.252+0000 7fea427fc640 1 -- 172.21.15.66:0/2301691360 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea2c003520 con 0x7fea440b4d60 2024-03-25T18:07:44.263 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.262+0000 7fea427fc640 1 -- 172.21.15.66:0/2301691360 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2673094811 0 0) 0x7fea2c003520 con 0x7fea440b4d60 2024-03-25T18:07:44.263 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.262+0000 7fea427fc640 1 -- 172.21.15.66:0/2301691360 >> v1:172.21.15.66:6790/0 conn(0x7fea440b1730 legacy=0x7fea440b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:44.263 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.262+0000 7fea427fc640 1 -- 172.21.15.66:0/2301691360 >> v1:172.21.15.193:6789/0 conn(0x7fea4414be90 legacy=0x7fea4414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:44.263 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.262+0000 7fea427fc640 10 monclient: _finish_hunting 0 2024-03-25T18:07:44.263 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.262+0000 7fea427fc640 1 monclient: found mon.noname-a 2024-03-25T18:07:44.264 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.262+0000 7fea427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:44.264 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.262+0000 7fea427fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.66:6789/0 2024-03-25T18:07:44.264 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.262+0000 7fea427fc640 1 -- 172.21.15.66:0/2301691360 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea440b1030 con 0x7fea440b4d60 2024-03-25T18:07:44.264 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.263+0000 7fea427fc640 10 monclient: _finish_auth 0 2024-03-25T18:07:44.264 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.263+0000 7fea427fc640 10 monclient: _check_auth_tickets 2024-03-25T18:07:44.264 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.263+0000 7fea427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:44.264 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.263+0000 7fea427fc640 1 -- 172.21.15.66:0/2301691360 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fea34001690 con 0x7fea440b4d60 2024-03-25T18:07:44.264 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.263+0000 7fea49eae640 5 monclient: authenticate success, global_id 4395 2024-03-25T18:07:44.264 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.263+0000 7fea49eae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:44.264 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.263+0000 7fea427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:44.264 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.263+0000 7fea427fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:07:44.264 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.263+0000 7fea427fc640 10 monclient: dump: 2024-03-25T18:07:44.264 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:44.264 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:44.264 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:44.265 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:44.265 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:44.265 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:44.265 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:44.265 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:44.265 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:44.265 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:44.265 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.263+0000 7fea49eae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:44.270 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.269+0000 7fea427fc640 1 -- 172.21.15.66:0/2301691360 <== mon.0 v1:172.21.15.66:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea340017f0 con 0x7fea440b4d60 2024-03-25T18:07:44.270 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.269+0000 7fea427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:44.270 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.269+0000 7fea427fc640 1 -- 172.21.15.66:0/2301691360 <== mon.0 v1:172.21.15.66:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fea34001c00 con 0x7fea440b4d60 2024-03-25T18:07:44.270 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.269+0000 7fea427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:44.270 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.269+0000 7fea427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:44.270 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.269+0000 7fea427fc640 10 monclient: dump: 2024-03-25T18:07:44.270 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:44.270 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:44.271 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:44.271 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:44.271 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:44.271 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:44.271 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:44.271 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:44.271 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:44.271 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:44.271 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.270+0000 7fea49eae640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:44.271 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.270+0000 7fea49eae640 4 set_mon_vals no callback set 2024-03-25T18:07:44.272 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.270+0000 7fea49eae640 10 monclient: shutdown 2024-03-25T18:07:44.272 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.270+0000 7fea49eae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:44.272 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.270+0000 7fea49eae640 1 -- 172.21.15.66:0/2301691360 >> v1:172.21.15.66:6789/0 conn(0x7fea440b4d60 legacy=0x7fea4413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:44.272 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.271+0000 7fea49eae640 1 -- 172.21.15.66:0/2301691360 shutdown_connections 2024-03-25T18:07:44.272 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.271+0000 7fea49eae640 1 -- 172.21.15.66:0/2301691360 >> 172.21.15.66:0/2301691360 conn(0x7fea440617a0 msgr2=0x7fea44061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:44.272 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.271+0000 7fea49eae640 1 -- 172.21.15.66:0/2301691360 shutdown_connections 2024-03-25T18:07:44.272 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.271+0000 7fea49eae640 1 -- 172.21.15.66:0/2301691360 wait complete. 2024-03-25T18:07:44.272 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.272+0000 7fea49eae640 10 monclient: build_initial_monmap 2024-03-25T18:07:44.273 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.272+0000 7fea49eae640 10 monclient: monmap: 2024-03-25T18:07:44.273 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:07:44.273 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:44.273 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:07:44.273 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:07:44.273 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:44.273 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:07:44.273 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:07:44.273 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:07:44.273 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:07:44.273 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:44.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.272+0000 7fea49eae640 1 librados: starting msgr at 2024-03-25T18:07:44.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.272+0000 7fea49eae640 1 librados: starting objecter 2024-03-25T18:07:44.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.273+0000 7fea49eae640 1 Processor -- start 2024-03-25T18:07:44.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.273+0000 7fea49eae640 1 -- start start 2024-03-25T18:07:44.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.273+0000 7fea49eae640 1 librados: setting wanted keys 2024-03-25T18:07:44.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.273+0000 7fea49eae640 1 librados: calling monclient init 2024-03-25T18:07:44.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.273+0000 7fea49eae640 10 monclient: init 2024-03-25T18:07:44.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.274+0000 7fea49eae640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:44.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.274+0000 7fea49eae640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:07:44.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.274+0000 7fea49eae640 10 monclient(hunting): picked mon.noname-a con 0x7fea440b1730 addr v1:172.21.15.66:6789/0 2024-03-25T18:07:44.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.274+0000 7fea49eae640 10 monclient(hunting): picked mon.noname-c con 0x7fea440b4d60 addr v1:172.21.15.66:6790/0 2024-03-25T18:07:44.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.274+0000 7fea49eae640 10 monclient(hunting): picked mon.noname-b con 0x7fea4414be90 addr v1:172.21.15.193:6789/0 2024-03-25T18:07:44.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.274+0000 7fea49eae640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4413ec20 con 0x7fea440b1730 2024-03-25T18:07:44.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.274+0000 7fea49eae640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4413ef00 con 0x7fea4414be90 2024-03-25T18:07:44.275 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.274+0000 7fea49eae640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea441452e0 con 0x7fea440b4d60 2024-03-25T18:07:44.276 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.274+0000 7fea49eae640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:44.276 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.274+0000 7fea49eae640 10 monclient(hunting): authenticate will time out at 7213.333456s 2024-03-25T18:07:44.287 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.287+0000 7fea43fff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fea4414be90 0x7fea441e25d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:45308/0 (socket says 172.21.15.66:45308) 2024-03-25T18:07:44.287 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.287+0000 7fea43fff640 1 -- 172.21.15.66:0/380371279 learned_addr learned my addr 172.21.15.66:0/380371279 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:07:44.292 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.292+0000 7fea2bfff640 10 client.?.objecter ms_handle_connect 0x7fea4414be90 2024-03-25T18:07:44.292 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.292+0000 7fea2bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:44.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.297+0000 7fea2bfff640 1 -- 172.21.15.66:0/380371279 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2984794569 0 0) 0x7fea4413ef00 con 0x7fea4414be90 2024-03-25T18:07:44.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.297+0000 7fea2bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:07:44.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.297+0000 7fea2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:44.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.297+0000 7fea2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:44.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.297+0000 7fea2bfff640 10 monclient(hunting): my global_id is 4393 2024-03-25T18:07:44.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.297+0000 7fea2bfff640 1 -- 172.21.15.66:0/380371279 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea30004eb0 con 0x7fea4414be90 2024-03-25T18:07:44.300 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.299+0000 7fea2bfff640 10 client.?.objecter ms_handle_connect 0x7fea440b1730 2024-03-25T18:07:44.300 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.299+0000 7fea2bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:44.306 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.306+0000 7fea2bfff640 1 -- 172.21.15.66:0/380371279 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958920647 0 0) 0x7fea4413ec20 con 0x7fea440b1730 2024-03-25T18:07:44.306 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.306+0000 7fea2bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:07:44.306 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.306+0000 7fea2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:44.307 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.306+0000 7fea2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:44.307 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.306+0000 7fea2bfff640 10 monclient(hunting): my global_id is 4398 2024-03-25T18:07:44.307 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.306+0000 7fea2bfff640 1 -- 172.21.15.66:0/380371279 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea4413ef00 con 0x7fea440b1730 2024-03-25T18:07:44.312 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.312+0000 7fea2bfff640 1 -- 172.21.15.66:0/380371279 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1678289598 0 0) 0x7fea30004eb0 con 0x7fea4414be90 2024-03-25T18:07:44.312 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.312+0000 7fea2bfff640 1 -- 172.21.15.66:0/380371279 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea4413ec20 con 0x7fea4414be90 2024-03-25T18:07:44.313 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.312+0000 7fea2bfff640 1 -- 172.21.15.66:0/380371279 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fea3c001c30 con 0x7fea4414be90 2024-03-25T18:07:44.313 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.312+0000 7fea2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:44.313 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.312+0000 7fea2bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:07:44.313 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.312+0000 7fea2bfff640 10 monclient(hunting): dump: 2024-03-25T18:07:44.313 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:44.313 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:44.313 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:44.313 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:44.313 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:44.313 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:44.313 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:44.313 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:44.313 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:44.313 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:44.314 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.312+0000 7fea2bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:44.317 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.317+0000 7fea2bfff640 10 client.?.objecter ms_handle_connect 0x7fea440b4d60 2024-03-25T18:07:44.317 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.317+0000 7fea2bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:44.330 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.330+0000 7fea2bfff640 1 -- 172.21.15.66:0/380371279 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4110755290 0 0) 0x7fea4413ef00 con 0x7fea440b1730 2024-03-25T18:07:44.330 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.330+0000 7fea2bfff640 1 -- 172.21.15.66:0/380371279 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea30004eb0 con 0x7fea440b1730 2024-03-25T18:07:44.330 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.330+0000 7fea2bfff640 1 -- 172.21.15.66:0/380371279 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fea34001bc0 con 0x7fea440b1730 2024-03-25T18:07:44.330 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.330+0000 7fea2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:44.330 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.330+0000 7fea2bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:44.330 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.330+0000 7fea2bfff640 10 monclient(hunting): dump: 2024-03-25T18:07:44.331 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:44.331 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:44.331 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:44.331 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:44.331 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:44.331 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:44.331 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:44.331 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:44.331 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:44.331 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:44.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.335+0000 7fea2bfff640 1 -- 172.21.15.66:0/380371279 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2183119191 0 0) 0x7fea4413ec20 con 0x7fea4414be90 2024-03-25T18:07:44.335 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.335+0000 7fea2bfff640 1 -- 172.21.15.66:0/380371279 >> v1:172.21.15.66:6790/0 conn(0x7fea440b4d60 legacy=0x7fea4413d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:44.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.335+0000 7fea2bfff640 1 -- 172.21.15.66:0/380371279 >> v1:172.21.15.66:6789/0 conn(0x7fea440b1730 legacy=0x7fea4413d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:44.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.335+0000 7fea2bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:07:44.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.335+0000 7fea2bfff640 1 monclient: found mon.b 2024-03-25T18:07:44.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.335+0000 7fea2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:44.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.335+0000 7fea2bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:07:44.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.335+0000 7fea2bfff640 1 -- 172.21.15.66:0/380371279 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea441455c0 con 0x7fea4414be90 2024-03-25T18:07:44.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.335+0000 7fea2bfff640 10 monclient: _finish_auth 0 2024-03-25T18:07:44.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.335+0000 7fea2bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:07:44.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.335+0000 7fea2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:44.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.335+0000 7fea49eae640 5 monclient: authenticate success, global_id 4393 2024-03-25T18:07:44.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.335+0000 7fea49eae640 10 monclient: _renew_subs 2024-03-25T18:07:44.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.335+0000 7fea49eae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:07:44.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.335+0000 7fea49eae640 1 -- 172.21.15.66:0/380371279 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea441458a0 con 0x7fea4414be90 2024-03-25T18:07:44.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.335+0000 7fea49eae640 10 client.4393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:44.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.335+0000 7fea49eae640 10 monclient: _renew_subs 2024-03-25T18:07:44.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.335+0000 7fea49eae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:07:44.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.335+0000 7fea49eae640 1 -- 172.21.15.66:0/380371279 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea44143380 con 0x7fea4414be90 2024-03-25T18:07:44.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.335+0000 7fea49eae640 1 librados: init done 2024-03-25T18:07:44.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.336+0000 7fea49eae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:44.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.336+0000 7fea49eae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:44.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.336+0000 7fea49eae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:07:44.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.336+0000 7fea49eae640 1 -- 172.21.15.66:0/380371279 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea440b3cd0 con 0x7fea4414be90 2024-03-25T18:07:44.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.622+0000 7fea2bfff640 1 -- 172.21.15.66:0/380371279 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea3c003990 con 0x7fea4414be90 2024-03-25T18:07:44.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.622+0000 7fea2bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:44.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.622+0000 7fea2bfff640 1 -- 172.21.15.66:0/380371279 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fea3c003d20 con 0x7fea4414be90 2024-03-25T18:07:44.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.622+0000 7fea2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:44.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.622+0000 7fea2bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:44.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.622+0000 7fea2bfff640 10 monclient: dump: 2024-03-25T18:07:44.623 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:44.624 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:44.624 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:44.624 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:44.624 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:44.624 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:44.624 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:44.624 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:44.624 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:44.624 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:44.624 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.622+0000 7fea41ffb640 4 set_mon_vals no callback set 2024-03-25T18:07:44.625 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.623+0000 7fea2bfff640 1 -- 172.21.15.66:0/380371279 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7fea3c003f00 con 0x7fea4414be90 2024-03-25T18:07:44.625 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.624+0000 7fea437fe640 10 monclient: get_auth_request con 0x7fea3006ddf0 auth_method 0 2024-03-25T18:07:44.625 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.625+0000 7fea2bfff640 10 client.4393.objecter ms_handle_connect 0x7fea3006ddf0 2024-03-25T18:07:44.628 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.628+0000 7fea2bfff640 1 -- 172.21.15.66:0/380371279 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7fea3c0549c0 con 0x7fea4414be90 2024-03-25T18:07:44.628 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.628+0000 7fea2bfff640 10 client.4393.objecter ms_dispatch 0x7fea44000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:07:44.628 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.628+0000 7fea2bfff640 3 client.4393.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:07:44.628 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.628+0000 7fea2bfff640 3 client.4393.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:07:44.628 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.628+0000 7fea2bfff640 20 client.4393.objecter dump_active .. 0 homeless 2024-03-25T18:07:44.629 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.628+0000 7fea2bfff640 1 -- 172.21.15.66:0/380371279 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fea3c0504d0 con 0x7fea4414be90 2024-03-25T18:07:44.629 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.628+0000 7fea2bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:44.629 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.628+0000 7fea2bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:44.878 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.877+0000 7fea49eae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:44.878 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.877+0000 7fea49eae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:44.878 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.877+0000 7fea49eae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:07:44.878 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.877+0000 7fea49eae640 1 -- 172.21.15.66:0/380371279 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea4413f0b0 con 0x7fea4414be90 2024-03-25T18:07:44.910 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.910+0000 7fea2bfff640 1 -- 172.21.15.66:0/380371279 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7fea3c052400 con 0x7fea4414be90 2024-03-25T18:07:44.911 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.910+0000 7fea2bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:44.911 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.910+0000 7fea2bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:44.911 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:07:44.911 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:07:44.914 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.913+0000 7fea49eae640 10 librados: watch_flush enter 2024-03-25T18:07:44.914 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.913+0000 7fea49eae640 10 librados: watch_flush exit 2024-03-25T18:07:44.914 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.913+0000 7fea49eae640 20 client.4393.objecter shutdown clearing up homeless session... 2024-03-25T18:07:44.914 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.913+0000 7fea49eae640 10 client.4393.objecter successfully canceled tick 2024-03-25T18:07:44.914 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.913+0000 7fea49eae640 1 -- 172.21.15.66:0/380371279 >> v1:172.21.15.193:6800/2275676803 conn(0x7fea3006ddf0 legacy=0x7fea300702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:44.914 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.913+0000 7fea49eae640 10 monclient: shutdown 2024-03-25T18:07:44.914 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.913+0000 7fea49eae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:44.915 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.913+0000 7fea49eae640 1 -- 172.21.15.66:0/380371279 >> v1:172.21.15.193:6789/0 conn(0x7fea4414be90 legacy=0x7fea441e25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:44.915 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.913+0000 7fea49eae640 1 -- 172.21.15.66:0/380371279 shutdown_connections 2024-03-25T18:07:44.915 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.913+0000 7fea49eae640 1 -- 172.21.15.66:0/380371279 >> 172.21.15.66:0/380371279 conn(0x7fea440617a0 msgr2=0x7fea440b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:44.915 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.914+0000 7fea49eae640 1 -- 172.21.15.66:0/380371279 shutdown_connections 2024-03-25T18:07:44.915 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.914+0000 7fea49eae640 1 -- 172.21.15.66:0/380371279 wait complete. 2024-03-25T18:07:44.915 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:44.914+0000 7fea49eae640 1 librados: shutdown 2024-03-25T18:07:44.926 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:07:51.733 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:07:51.853 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.853+0000 7fcfffc27640 10 monclient: get_monmap_and_config 2024-03-25T18:07:51.853 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.853+0000 7fcfffc27640 10 monclient: build_initial_monmap 2024-03-25T18:07:51.853 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.853+0000 7fcfffc27640 10 monclient: monmap: 2024-03-25T18:07:51.854 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:07:51.854 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:51.854 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:07:51.853700+0000 2024-03-25T18:07:51.854 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:07:51.853700+0000 2024-03-25T18:07:51.854 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:51.854 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:51.854 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:07:51.854 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:07:51.854 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:07:51.854 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:51.855 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.853+0000 7fcfffc27640 1 Processor -- start 2024-03-25T18:07:51.855 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.854+0000 7fcfffc27640 1 -- start start 2024-03-25T18:07:51.855 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.854+0000 7fcfffc27640 10 monclient: init 2024-03-25T18:07:51.855 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.854+0000 7fcfffc27640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:51.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.854+0000 7fcfffc27640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:07:51.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.854+0000 7fcfffc27640 10 monclient(hunting): picked mon.noname-c con 0x7fcff80b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T18:07:51.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.854+0000 7fcfffc27640 10 monclient(hunting): picked mon.noname-b con 0x7fcff80b1730 addr v1:172.21.15.66:6790/0 2024-03-25T18:07:51.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.854+0000 7fcfffc27640 10 monclient(hunting): picked mon.noname-a con 0x7fcff814be90 addr v1:172.21.15.66:6789/0 2024-03-25T18:07:51.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.855+0000 7fcfffc27640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff80af940 con 0x7fcff814be90 2024-03-25T18:07:51.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.855+0000 7fcfffc27640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff80b0a70 con 0x7fcff80b4d60 2024-03-25T18:07:51.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.855+0000 7fcfffc27640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff80b0d50 con 0x7fcff80b1730 2024-03-25T18:07:51.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.855+0000 7fcfffc27640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:51.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.855+0000 7fcfffc27640 10 monclient(hunting): authenticate will time out at 7220.914152s 2024-03-25T18:07:51.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.856+0000 7fcffe1c4640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7fcff814be90 0x7fcff814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:38556/0 (socket says 172.21.15.66:38556) 2024-03-25T18:07:51.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.856+0000 7fcffe1c4640 1 -- 172.21.15.66:0/1568220773 learned_addr learned my addr 172.21.15.66:0/1568220773 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:07:51.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.867+0000 7fcffc9c1640 1 -- 172.21.15.66:0/1568220773 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721759796 0 0) 0x7fcff80af940 con 0x7fcff814be90 2024-03-25T18:07:51.868 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.867+0000 7fcffc9c1640 10 monclient(hunting): _negotiate 2024-03-25T18:07:51.868 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.867+0000 7fcffc9c1640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:51.868 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.867+0000 7fcffc9c1640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:51.868 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.867+0000 7fcffc9c1640 10 monclient(hunting): my global_id is 4401 2024-03-25T18:07:51.868 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.867+0000 7fcffc9c1640 1 -- 172.21.15.66:0/1568220773 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfe4003520 con 0x7fcff814be90 2024-03-25T18:07:51.880 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.879+0000 7fcffc9c1640 1 -- 172.21.15.66:0/1568220773 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2959813673 0 0) 0x7fcfe4003520 con 0x7fcff814be90 2024-03-25T18:07:51.880 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.879+0000 7fcffc9c1640 1 -- 172.21.15.66:0/1568220773 >> v1:172.21.15.66:6790/0 conn(0x7fcff80b1730 legacy=0x7fcff80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:51.880 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.879+0000 7fcffc9c1640 1 -- 172.21.15.66:0/1568220773 >> v1:172.21.15.193:6789/0 conn(0x7fcff80b4d60 legacy=0x7fcff813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:51.880 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.879+0000 7fcffc9c1640 10 monclient: _finish_hunting 0 2024-03-25T18:07:51.880 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.879+0000 7fcffc9c1640 1 monclient: found mon.noname-a 2024-03-25T18:07:51.880 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.879+0000 7fcffc9c1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:51.880 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.879+0000 7fcffc9c1640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.66:6789/0 2024-03-25T18:07:51.880 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.879+0000 7fcffc9c1640 1 -- 172.21.15.66:0/1568220773 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff80b1030 con 0x7fcff814be90 2024-03-25T18:07:51.880 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.879+0000 7fcffc9c1640 10 monclient: _finish_auth 0 2024-03-25T18:07:51.880 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.879+0000 7fcffc9c1640 10 monclient: _check_auth_tickets 2024-03-25T18:07:51.880 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.879+0000 7fcffc9c1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:51.880 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.879+0000 7fcffc9c1640 1 -- 172.21.15.66:0/1568220773 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fcff0001a10 con 0x7fcff814be90 2024-03-25T18:07:51.880 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.879+0000 7fcffc9c1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:51.880 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.879+0000 7fcffc9c1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:07:51.881 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.879+0000 7fcffc9c1640 10 monclient: dump: 2024-03-25T18:07:51.881 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:51.881 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:51.881 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:51.881 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:51.881 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:51.881 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:51.881 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:51.881 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:51.881 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:51.881 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:51.881 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.879+0000 7fcfffc27640 5 monclient: authenticate success, global_id 4401 2024-03-25T18:07:51.881 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.879+0000 7fcfffc27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:51.886 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.886+0000 7fcffc9c1640 1 -- 172.21.15.66:0/1568220773 <== mon.0 v1:172.21.15.66:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcff0001b70 con 0x7fcff814be90 2024-03-25T18:07:51.886 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.886+0000 7fcffc9c1640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:51.886 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.886+0000 7fcffc9c1640 1 -- 172.21.15.66:0/1568220773 <== mon.0 v1:172.21.15.66:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fcff0001f80 con 0x7fcff814be90 2024-03-25T18:07:51.886 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.886+0000 7fcfffc27640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:51.886 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.886+0000 7fcfffc27640 4 set_mon_vals no callback set 2024-03-25T18:07:51.886 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.886+0000 7fcffc9c1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:51.886 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.886+0000 7fcffc9c1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:51.886 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.886+0000 7fcffc9c1640 10 monclient: dump: 2024-03-25T18:07:51.886 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:51.887 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:51.887 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:51.887 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:51.887 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:51.887 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:51.887 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:51.887 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:51.887 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:51.887 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:51.889 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.886+0000 7fcfffc27640 10 monclient: shutdown 2024-03-25T18:07:51.889 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.886+0000 7fcfffc27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:51.889 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.886+0000 7fcfffc27640 1 -- 172.21.15.66:0/1568220773 >> v1:172.21.15.66:6789/0 conn(0x7fcff814be90 legacy=0x7fcff814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:51.889 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.886+0000 7fcfffc27640 1 -- 172.21.15.66:0/1568220773 shutdown_connections 2024-03-25T18:07:51.889 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.886+0000 7fcfffc27640 1 -- 172.21.15.66:0/1568220773 >> 172.21.15.66:0/1568220773 conn(0x7fcff80617a0 msgr2=0x7fcff8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:51.890 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.887+0000 7fcfffc27640 1 -- 172.21.15.66:0/1568220773 shutdown_connections 2024-03-25T18:07:51.890 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.887+0000 7fcfffc27640 1 -- 172.21.15.66:0/1568220773 wait complete. 2024-03-25T18:07:51.890 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.887+0000 7fcfffc27640 10 monclient: build_initial_monmap 2024-03-25T18:07:51.890 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.887+0000 7fcfffc27640 10 monclient: monmap: 2024-03-25T18:07:51.890 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:07:51.890 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:51.890 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:07:51.890 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:07:51.890 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:51.890 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:07:51.890 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:07:51.890 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:07:51.890 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:07:51.890 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:51.891 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.888+0000 7fcfffc27640 1 librados: starting msgr at 2024-03-25T18:07:51.891 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.888+0000 7fcfffc27640 1 librados: starting objecter 2024-03-25T18:07:51.891 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.888+0000 7fcfffc27640 1 Processor -- start 2024-03-25T18:07:51.891 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.888+0000 7fcfffc27640 1 -- start start 2024-03-25T18:07:51.891 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.888+0000 7fcfffc27640 1 librados: setting wanted keys 2024-03-25T18:07:51.891 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.888+0000 7fcfffc27640 1 librados: calling monclient init 2024-03-25T18:07:51.891 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.888+0000 7fcfffc27640 10 monclient: init 2024-03-25T18:07:51.891 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.888+0000 7fcfffc27640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:51.891 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.888+0000 7fcfffc27640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:07:51.891 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.888+0000 7fcfffc27640 10 monclient(hunting): picked mon.noname-c con 0x7fcff80b1730 addr v1:172.21.15.66:6790/0 2024-03-25T18:07:51.891 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.888+0000 7fcfffc27640 10 monclient(hunting): picked mon.noname-b con 0x7fcff80b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T18:07:51.891 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.889+0000 7fcfffc27640 10 monclient(hunting): picked mon.noname-a con 0x7fcff814be90 addr v1:172.21.15.66:6789/0 2024-03-25T18:07:51.891 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.889+0000 7fcfffc27640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff813bd10 con 0x7fcff814be90 2024-03-25T18:07:51.891 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.889+0000 7fcfffc27640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff81dadb0 con 0x7fcff80b4d60 2024-03-25T18:07:51.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.889+0000 7fcfffc27640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcff8138950 con 0x7fcff80b1730 2024-03-25T18:07:51.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.889+0000 7fcfffc27640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:51.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.889+0000 7fcfffc27640 10 monclient(hunting): authenticate will time out at 7220.948369s 2024-03-25T18:07:51.900 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.899+0000 7fcffd1c2640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fcff80b4d60 0x7fcff813a450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:49722/0 (socket says 172.21.15.66:49722) 2024-03-25T18:07:51.900 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.899+0000 7fcffd1c2640 1 -- 172.21.15.66:0/3602580888 learned_addr learned my addr 172.21.15.66:0/3602580888 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:07:51.900 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.900+0000 7fcfee7fc640 10 client.?.objecter ms_handle_connect 0x7fcff80b4d60 2024-03-25T18:07:51.900 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.900+0000 7fcfee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:51.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.908+0000 7fcfee7fc640 10 client.?.objecter ms_handle_connect 0x7fcff814be90 2024-03-25T18:07:51.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.908+0000 7fcfee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:51.910 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.909+0000 7fcfee7fc640 10 client.?.objecter ms_handle_connect 0x7fcff80b1730 2024-03-25T18:07:51.910 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.909+0000 7fcfee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:51.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.918+0000 7fcfee7fc640 1 -- 172.21.15.66:0/3602580888 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4187178783 0 0) 0x7fcff8138950 con 0x7fcff80b1730 2024-03-25T18:07:51.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.918+0000 7fcfee7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:07:51.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.918+0000 7fcfee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:51.919 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.918+0000 7fcfee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:51.919 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.918+0000 7fcfee7fc640 10 monclient(hunting): my global_id is 4403 2024-03-25T18:07:51.919 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.918+0000 7fcfee7fc640 1 -- 172.21.15.66:0/3602580888 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfdc004eb0 con 0x7fcff80b1730 2024-03-25T18:07:51.921 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.920+0000 7fcfee7fc640 1 -- 172.21.15.66:0/3602580888 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942213327 0 0) 0x7fcff81dadb0 con 0x7fcff80b4d60 2024-03-25T18:07:51.921 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.920+0000 7fcfee7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:07:51.921 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.920+0000 7fcfee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:51.921 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.920+0000 7fcfee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:51.921 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.920+0000 7fcfee7fc640 10 monclient(hunting): my global_id is 4399 2024-03-25T18:07:51.921 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.920+0000 7fcfee7fc640 1 -- 172.21.15.66:0/3602580888 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcff8138950 con 0x7fcff80b4d60 2024-03-25T18:07:51.930 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.929+0000 7fcfee7fc640 1 -- 172.21.15.66:0/3602580888 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3861443170 0 0) 0x7fcff813bd10 con 0x7fcff814be90 2024-03-25T18:07:51.930 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.929+0000 7fcfee7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:07:51.930 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.929+0000 7fcfee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:51.930 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.929+0000 7fcfee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:51.930 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.929+0000 7fcfee7fc640 10 monclient(hunting): my global_id is 4404 2024-03-25T18:07:51.930 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.929+0000 7fcfee7fc640 1 -- 172.21.15.66:0/3602580888 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcff81dadb0 con 0x7fcff814be90 2024-03-25T18:07:51.934 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.934+0000 7fcfee7fc640 1 -- 172.21.15.66:0/3602580888 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 475088259 0 0) 0x7fcfdc004eb0 con 0x7fcff80b1730 2024-03-25T18:07:51.935 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.934+0000 7fcfee7fc640 1 -- 172.21.15.66:0/3602580888 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcff813bd10 con 0x7fcff80b1730 2024-03-25T18:07:51.935 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.934+0000 7fcfee7fc640 1 -- 172.21.15.66:0/3602580888 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fcff4001930 con 0x7fcff80b1730 2024-03-25T18:07:51.935 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.934+0000 7fcfee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:51.935 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.934+0000 7fcfee7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:07:51.935 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.934+0000 7fcfee7fc640 10 monclient(hunting): dump: 2024-03-25T18:07:51.935 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:51.935 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:51.935 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:51.935 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:51.935 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:51.935 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:51.935 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:51.935 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:51.936 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:51.936 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:51.936 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.934+0000 7fcfee7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:51.938 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.938+0000 7fcfee7fc640 1 -- 172.21.15.66:0/3602580888 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1006266695 0 0) 0x7fcff8138950 con 0x7fcff80b4d60 2024-03-25T18:07:51.939 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.938+0000 7fcfee7fc640 1 -- 172.21.15.66:0/3602580888 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcfdc004eb0 con 0x7fcff80b4d60 2024-03-25T18:07:51.939 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.938+0000 7fcfee7fc640 1 -- 172.21.15.66:0/3602580888 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fcfe80019b0 con 0x7fcff80b4d60 2024-03-25T18:07:51.939 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.938+0000 7fcfee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:51.939 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.938+0000 7fcfee7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:51.939 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.938+0000 7fcfee7fc640 10 monclient(hunting): dump: 2024-03-25T18:07:51.939 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:51.939 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:51.939 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:51.939 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:51.939 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:51.939 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:51.939 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:51.939 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:51.939 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:51.939 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:51.947 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.946+0000 7fcfee7fc640 1 -- 172.21.15.66:0/3602580888 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2475724541 0 0) 0x7fcff81dadb0 con 0x7fcff814be90 2024-03-25T18:07:51.947 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.946+0000 7fcfee7fc640 1 -- 172.21.15.66:0/3602580888 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcff8138950 con 0x7fcff814be90 2024-03-25T18:07:51.947 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.946+0000 7fcfee7fc640 1 -- 172.21.15.66:0/3602580888 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fcff0001e80 con 0x7fcff814be90 2024-03-25T18:07:51.947 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.946+0000 7fcfee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:51.947 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.946+0000 7fcfee7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:51.947 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.946+0000 7fcfee7fc640 10 monclient(hunting): dump: 2024-03-25T18:07:51.947 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:51.947 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:51.947 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:51.947 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:51.947 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:51.947 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:51.947 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:51.947 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:51.947 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:51.948 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:51.953 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.952+0000 7fcfee7fc640 1 -- 172.21.15.66:0/3602580888 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3092820376 0 0) 0x7fcfdc004eb0 con 0x7fcff80b4d60 2024-03-25T18:07:51.953 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.952+0000 7fcfee7fc640 1 -- 172.21.15.66:0/3602580888 >> v1:172.21.15.66:6790/0 conn(0x7fcff80b1730 legacy=0x7fcff8138240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:51.953 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.952+0000 7fcfee7fc640 1 -- 172.21.15.66:0/3602580888 >> v1:172.21.15.66:6789/0 conn(0x7fcff814be90 legacy=0x7fcff81db2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:51.953 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.952+0000 7fcfee7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:07:51.953 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.952+0000 7fcfee7fc640 1 monclient: found mon.b 2024-03-25T18:07:51.954 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.952+0000 7fcfee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:51.954 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.952+0000 7fcfee7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:07:51.954 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.952+0000 7fcfee7fc640 1 -- 172.21.15.66:0/3602580888 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff813ab60 con 0x7fcff80b4d60 2024-03-25T18:07:51.954 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.952+0000 7fcfee7fc640 10 monclient: _finish_auth 0 2024-03-25T18:07:51.954 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.952+0000 7fcfee7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:07:51.954 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.952+0000 7fcfee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:51.954 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.952+0000 7fcfffc27640 5 monclient: authenticate success, global_id 4399 2024-03-25T18:07:51.954 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.952+0000 7fcfffc27640 10 monclient: _renew_subs 2024-03-25T18:07:51.954 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.952+0000 7fcfffc27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:07:51.954 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.952+0000 7fcfffc27640 1 -- 172.21.15.66:0/3602580888 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcff81db9e0 con 0x7fcff80b4d60 2024-03-25T18:07:51.954 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.952+0000 7fcfffc27640 10 client.4399.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:51.954 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.952+0000 7fcfffc27640 10 monclient: _renew_subs 2024-03-25T18:07:51.954 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.952+0000 7fcfffc27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:07:51.954 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.952+0000 7fcfffc27640 1 -- 172.21.15.66:0/3602580888 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcff8139890 con 0x7fcff80b4d60 2024-03-25T18:07:51.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.952+0000 7fcfffc27640 1 librados: init done 2024-03-25T18:07:51.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.953+0000 7fcfffc27640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:51.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.953+0000 7fcfffc27640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:51.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.953+0000 7fcfffc27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:07:51.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:51.953+0000 7fcfffc27640 1 -- 172.21.15.66:0/3602580888 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcff813acf0 con 0x7fcff80b4d60 2024-03-25T18:07:52.229 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.229+0000 7fcfee7fc640 1 -- 172.21.15.66:0/3602580888 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcfe80035f0 con 0x7fcff80b4d60 2024-03-25T18:07:52.230 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.229+0000 7fcfee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:52.230 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.229+0000 7fcfee7fc640 1 -- 172.21.15.66:0/3602580888 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fcfe8003980 con 0x7fcff80b4d60 2024-03-25T18:07:52.230 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.229+0000 7fcfee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:52.230 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.229+0000 7fcfee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:52.230 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.229+0000 7fcfee7fc640 10 monclient: dump: 2024-03-25T18:07:52.230 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:52.230 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:52.230 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:52.230 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:52.230 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:52.230 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:52.230 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:52.230 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:52.231 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:52.231 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:52.231 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.229+0000 7fcfeffff640 4 set_mon_vals no callback set 2024-03-25T18:07:52.231 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.229+0000 7fcfee7fc640 1 -- 172.21.15.66:0/3602580888 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7fcfe8003ba0 con 0x7fcff80b4d60 2024-03-25T18:07:52.231 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.230+0000 7fcffd9c3640 10 monclient: get_auth_request con 0x7fcfdc06e160 auth_method 0 2024-03-25T18:07:52.231 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.230+0000 7fcfee7fc640 10 client.4399.objecter ms_handle_connect 0x7fcfdc06e160 2024-03-25T18:07:52.236 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.235+0000 7fcfee7fc640 1 -- 172.21.15.66:0/3602580888 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7fcfe80547a0 con 0x7fcff80b4d60 2024-03-25T18:07:52.236 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.235+0000 7fcfee7fc640 10 client.4399.objecter ms_dispatch 0x7fcff8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:07:52.236 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.235+0000 7fcfee7fc640 3 client.4399.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:07:52.236 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.235+0000 7fcfee7fc640 3 client.4399.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:07:52.236 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.236+0000 7fcfee7fc640 20 client.4399.objecter dump_active .. 0 homeless 2024-03-25T18:07:52.238 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.238+0000 7fcfee7fc640 1 -- 172.21.15.66:0/3602580888 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcfe80502b0 con 0x7fcff80b4d60 2024-03-25T18:07:52.238 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.238+0000 7fcfee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:52.238 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.238+0000 7fcfee7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:52.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.467+0000 7fcfffc27640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:52.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.467+0000 7fcfffc27640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:52.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.467+0000 7fcfffc27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:07:52.468 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.467+0000 7fcfffc27640 1 -- 172.21.15.66:0/3602580888 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcff81dafa0 con 0x7fcff80b4d60 2024-03-25T18:07:52.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.496+0000 7fcfee7fc640 1 -- 172.21.15.66:0/3602580888 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7fcfe80521e0 con 0x7fcff80b4d60 2024-03-25T18:07:52.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.496+0000 7fcfee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:52.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.496+0000 7fcfee7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:52.497 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:07:52.497 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:07:52.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.499+0000 7fcfffc27640 10 librados: watch_flush enter 2024-03-25T18:07:52.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.499+0000 7fcfffc27640 10 librados: watch_flush exit 2024-03-25T18:07:52.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.499+0000 7fcfffc27640 20 client.4399.objecter shutdown clearing up homeless session... 2024-03-25T18:07:52.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.499+0000 7fcfffc27640 10 client.4399.objecter successfully canceled tick 2024-03-25T18:07:52.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.499+0000 7fcfffc27640 1 -- 172.21.15.66:0/3602580888 >> v1:172.21.15.193:6800/2275676803 conn(0x7fcfdc06e160 legacy=0x7fcfdc070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:52.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.499+0000 7fcfffc27640 10 monclient: shutdown 2024-03-25T18:07:52.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.499+0000 7fcfffc27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:52.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.499+0000 7fcfffc27640 1 -- 172.21.15.66:0/3602580888 >> v1:172.21.15.193:6789/0 conn(0x7fcff80b4d60 legacy=0x7fcff813a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:52.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.499+0000 7fcfffc27640 1 -- 172.21.15.66:0/3602580888 shutdown_connections 2024-03-25T18:07:52.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.499+0000 7fcfffc27640 1 -- 172.21.15.66:0/3602580888 >> 172.21.15.66:0/3602580888 conn(0x7fcff80617a0 msgr2=0x7fcff813c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:52.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.499+0000 7fcfffc27640 1 -- 172.21.15.66:0/3602580888 shutdown_connections 2024-03-25T18:07:52.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.499+0000 7fcfffc27640 1 -- 172.21.15.66:0/3602580888 wait complete. 2024-03-25T18:07:52.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:52.499+0000 7fcfffc27640 1 librados: shutdown 2024-03-25T18:07:52.512 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:07:59.318 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:07:59.433 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.432+0000 7f6b3b1ba640 10 monclient: get_monmap_and_config 2024-03-25T18:07:59.433 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.432+0000 7f6b3b1ba640 10 monclient: build_initial_monmap 2024-03-25T18:07:59.433 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.432+0000 7f6b3b1ba640 10 monclient: monmap: 2024-03-25T18:07:59.433 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:07:59.433 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:59.434 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:07:59.433457+0000 2024-03-25T18:07:59.434 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:07:59.433457+0000 2024-03-25T18:07:59.434 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:59.434 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:59.434 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:07:59.434 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:07:59.434 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:07:59.434 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:59.435 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.433+0000 7f6b3b1ba640 1 Processor -- start 2024-03-25T18:07:59.435 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.434+0000 7f6b3b1ba640 1 -- start start 2024-03-25T18:07:59.435 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.434+0000 7f6b3b1ba640 10 monclient: init 2024-03-25T18:07:59.435 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.434+0000 7f6b3b1ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:59.435 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.434+0000 7f6b3b1ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:07:59.435 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.434+0000 7f6b3b1ba640 10 monclient(hunting): picked mon.noname-c con 0x7f6b340a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:07:59.435 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.434+0000 7f6b3b1ba640 10 monclient(hunting): picked mon.noname-b con 0x7f6b3413f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:07:59.436 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.434+0000 7f6b3b1ba640 10 monclient(hunting): picked mon.noname-a con 0x7f6b341430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:07:59.436 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.435+0000 7f6b3b1ba640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b34142520 con 0x7f6b341430c0 2024-03-25T18:07:59.436 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.435+0000 7f6b3b1ba640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b340b28a0 con 0x7f6b340a98f0 2024-03-25T18:07:59.436 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.435+0000 7f6b3b1ba640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b340b2a30 con 0x7f6b3413f710 2024-03-25T18:07:59.436 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.435+0000 7f6b3b1ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:59.436 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.435+0000 7f6b3b1ba640 10 monclient(hunting): authenticate will time out at 7228.494166s 2024-03-25T18:07:59.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.447+0000 7f6b38f56640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f6b340a98f0 0x7f6b340a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:51654/0 (socket says 172.21.15.66:51654) 2024-03-25T18:07:59.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.447+0000 7f6b38f56640 1 -- 172.21.15.66:0/305969265 learned_addr learned my addr 172.21.15.66:0/305969265 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:07:59.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.453+0000 7f6b337fe640 1 -- 172.21.15.66:0/305969265 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2007359527 0 0) 0x7f6b340b28a0 con 0x7f6b340a98f0 2024-03-25T18:07:59.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.453+0000 7f6b337fe640 10 monclient(hunting): _negotiate 2024-03-25T18:07:59.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.453+0000 7f6b337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:59.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.453+0000 7f6b337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:59.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.453+0000 7f6b337fe640 10 monclient(hunting): my global_id is 4402 2024-03-25T18:07:59.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.453+0000 7f6b337fe640 1 -- 172.21.15.66:0/305969265 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b24003520 con 0x7f6b340a98f0 2024-03-25T18:07:59.457 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.457+0000 7f6b337fe640 1 -- 172.21.15.66:0/305969265 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3480675149 0 0) 0x7f6b340b2a30 con 0x7f6b3413f710 2024-03-25T18:07:59.458 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.457+0000 7f6b337fe640 10 monclient(hunting): _negotiate 2024-03-25T18:07:59.458 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.457+0000 7f6b337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:59.458 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.457+0000 7f6b337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:59.458 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.457+0000 7f6b337fe640 10 monclient(hunting): my global_id is 4406 2024-03-25T18:07:59.458 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.457+0000 7f6b337fe640 1 -- 172.21.15.66:0/305969265 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b340b28a0 con 0x7f6b3413f710 2024-03-25T18:07:59.464 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.464+0000 7f6b337fe640 1 -- 172.21.15.66:0/305969265 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2732693087 0 0) 0x7f6b340b28a0 con 0x7f6b3413f710 2024-03-25T18:07:59.464 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.464+0000 7f6b337fe640 1 -- 172.21.15.66:0/305969265 >> v1:172.21.15.193:6789/0 conn(0x7f6b340a98f0 legacy=0x7f6b340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:59.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.464+0000 7f6b337fe640 1 -- 172.21.15.66:0/305969265 >> v1:172.21.15.66:6789/0 conn(0x7f6b341430c0 legacy=0x7f6b3414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:07:59.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.464+0000 7f6b337fe640 10 monclient: _finish_hunting 0 2024-03-25T18:07:59.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.464+0000 7f6b337fe640 1 monclient: found mon.noname-b 2024-03-25T18:07:59.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.464+0000 7f6b337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:59.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.464+0000 7f6b337fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.66:6790/0 2024-03-25T18:07:59.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.464+0000 7f6b337fe640 1 -- 172.21.15.66:0/305969265 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b340b44c0 con 0x7f6b3413f710 2024-03-25T18:07:59.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.464+0000 7f6b337fe640 10 monclient: _finish_auth 0 2024-03-25T18:07:59.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.464+0000 7f6b337fe640 10 monclient: _check_auth_tickets 2024-03-25T18:07:59.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.464+0000 7f6b337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:59.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.464+0000 7f6b337fe640 1 -- 172.21.15.66:0/305969265 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f6b20001670 con 0x7f6b3413f710 2024-03-25T18:07:59.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.464+0000 7f6b3b1ba640 5 monclient: authenticate success, global_id 4406 2024-03-25T18:07:59.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.464+0000 7f6b3b1ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:59.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.464+0000 7f6b337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:59.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.464+0000 7f6b337fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:07:59.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.464+0000 7f6b337fe640 10 monclient: dump: 2024-03-25T18:07:59.466 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:59.466 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:59.466 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:59.466 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:59.466 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:59.466 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:59.466 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:59.466 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:59.466 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:59.466 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:59.466 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.464+0000 7f6b3b1ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:59.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.490+0000 7f6b337fe640 1 -- 172.21.15.66:0/305969265 <== mon.2 v1:172.21.15.66:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b200017d0 con 0x7f6b3413f710 2024-03-25T18:07:59.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.490+0000 7f6b337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:59.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.490+0000 7f6b337fe640 1 -- 172.21.15.66:0/305969265 <== mon.2 v1:172.21.15.66:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f6b20001c20 con 0x7f6b3413f710 2024-03-25T18:07:59.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.490+0000 7f6b3b1ba640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:59.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.490+0000 7f6b3b1ba640 4 set_mon_vals no callback set 2024-03-25T18:07:59.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.490+0000 7f6b337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:59.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.490+0000 7f6b337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:59.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.490+0000 7f6b337fe640 10 monclient: dump: 2024-03-25T18:07:59.491 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:59.491 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:59.491 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:59.491 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:59.491 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:59.491 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:59.491 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:59.491 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:59.492 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:59.492 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:59.493 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.491+0000 7f6b3b1ba640 10 monclient: shutdown 2024-03-25T18:07:59.493 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.491+0000 7f6b3b1ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:59.493 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.491+0000 7f6b3b1ba640 1 -- 172.21.15.66:0/305969265 >> v1:172.21.15.66:6790/0 conn(0x7f6b3413f710 legacy=0x7f6b341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:59.493 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.491+0000 7f6b3b1ba640 1 -- 172.21.15.66:0/305969265 shutdown_connections 2024-03-25T18:07:59.493 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.491+0000 7f6b3b1ba640 1 -- 172.21.15.66:0/305969265 >> 172.21.15.66:0/305969265 conn(0x7f6b340b7ea0 msgr2=0x7f6b340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:59.493 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.491+0000 7f6b3b1ba640 1 -- 172.21.15.66:0/305969265 shutdown_connections 2024-03-25T18:07:59.493 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.491+0000 7f6b3b1ba640 1 -- 172.21.15.66:0/305969265 wait complete. 2024-03-25T18:07:59.493 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.492+0000 7f6b3b1ba640 10 monclient: build_initial_monmap 2024-03-25T18:07:59.493 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.492+0000 7f6b3b1ba640 10 monclient: monmap: 2024-03-25T18:07:59.493 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:07:59.493 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:59.493 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:07:59.494 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:07:59.494 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:59.494 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:07:59.494 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:07:59.494 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:07:59.494 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:07:59.494 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:59.494 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.492+0000 7f6b3b1ba640 1 librados: starting msgr at 2024-03-25T18:07:59.494 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.492+0000 7f6b3b1ba640 1 librados: starting objecter 2024-03-25T18:07:59.494 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.493+0000 7f6b3b1ba640 1 Processor -- start 2024-03-25T18:07:59.494 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.493+0000 7f6b3b1ba640 1 -- start start 2024-03-25T18:07:59.494 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.493+0000 7f6b3b1ba640 1 librados: setting wanted keys 2024-03-25T18:07:59.494 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.493+0000 7f6b3b1ba640 1 librados: calling monclient init 2024-03-25T18:07:59.494 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.493+0000 7f6b3b1ba640 10 monclient: init 2024-03-25T18:07:59.495 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.493+0000 7f6b3b1ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:59.495 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.493+0000 7f6b3b1ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:07:59.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.493+0000 7f6b3b1ba640 10 monclient(hunting): picked mon.noname-b con 0x7f6b340a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:07:59.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.493+0000 7f6b3b1ba640 10 monclient(hunting): picked mon.noname-a con 0x7f6b3413f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:07:59.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.494+0000 7f6b3b1ba640 10 monclient(hunting): picked mon.noname-c con 0x7f6b341430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:07:59.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.494+0000 7f6b3b1ba640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b341db610 con 0x7f6b3413f710 2024-03-25T18:07:59.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.494+0000 7f6b3b1ba640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b340b03f0 con 0x7f6b340a98f0 2024-03-25T18:07:59.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.494+0000 7f6b3b1ba640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b340b06d0 con 0x7f6b341430c0 2024-03-25T18:07:59.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.494+0000 7f6b3b1ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:59.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.494+0000 7f6b3b1ba640 10 monclient(hunting): authenticate will time out at 7228.553155s 2024-03-25T18:07:59.499 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.499+0000 7f6b39757640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f6b341430c0 0x7f6b341daf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:58852/0 (socket says 172.21.15.66:58852) 2024-03-25T18:07:59.499 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.499+0000 7f6b39757640 1 -- 172.21.15.66:0/119342878 learned_addr learned my addr 172.21.15.66:0/119342878 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:07:59.503 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.503+0000 7f6b317fa640 10 client.?.objecter ms_handle_connect 0x7f6b341430c0 2024-03-25T18:07:59.503 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.503+0000 7f6b317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:59.506 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.506+0000 7f6b317fa640 10 client.?.objecter ms_handle_connect 0x7f6b340a98f0 2024-03-25T18:07:59.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.506+0000 7f6b317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:59.512 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.512+0000 7f6b317fa640 1 -- 172.21.15.66:0/119342878 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1951395491 0 0) 0x7f6b340b06d0 con 0x7f6b341430c0 2024-03-25T18:07:59.513 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.512+0000 7f6b317fa640 10 monclient(hunting): _negotiate 2024-03-25T18:07:59.513 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.512+0000 7f6b317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:59.513 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.512+0000 7f6b317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:59.513 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.512+0000 7f6b317fa640 10 monclient(hunting): my global_id is 4409 2024-03-25T18:07:59.513 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.512+0000 7f6b317fa640 1 -- 172.21.15.66:0/119342878 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b18004eb0 con 0x7f6b341430c0 2024-03-25T18:07:59.515 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.515+0000 7f6b317fa640 10 client.?.objecter ms_handle_connect 0x7f6b3413f710 2024-03-25T18:07:59.515 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.515+0000 7f6b317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:59.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.515+0000 7f6b317fa640 1 -- 172.21.15.66:0/119342878 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1054205946 0 0) 0x7f6b340b03f0 con 0x7f6b340a98f0 2024-03-25T18:07:59.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.515+0000 7f6b317fa640 10 monclient(hunting): _negotiate 2024-03-25T18:07:59.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.515+0000 7f6b317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:59.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.515+0000 7f6b317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:59.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.515+0000 7f6b317fa640 10 monclient(hunting): my global_id is 4405 2024-03-25T18:07:59.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.515+0000 7f6b317fa640 1 -- 172.21.15.66:0/119342878 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b340b06d0 con 0x7f6b340a98f0 2024-03-25T18:07:59.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.520+0000 7f6b317fa640 1 -- 172.21.15.66:0/119342878 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3325043949 0 0) 0x7f6b18004eb0 con 0x7f6b341430c0 2024-03-25T18:07:59.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.521+0000 7f6b317fa640 1 -- 172.21.15.66:0/119342878 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b340b03f0 con 0x7f6b341430c0 2024-03-25T18:07:59.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.521+0000 7f6b317fa640 1 -- 172.21.15.66:0/119342878 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f6b2c001b90 con 0x7f6b341430c0 2024-03-25T18:07:59.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.521+0000 7f6b317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:59.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.521+0000 7f6b317fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:07:59.522 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.521+0000 7f6b317fa640 10 monclient(hunting): dump: 2024-03-25T18:07:59.522 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:59.522 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:59.522 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:59.522 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:59.522 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:59.522 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:59.522 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:59.522 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:59.522 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:59.522 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:59.522 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.521+0000 7f6b317fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:59.526 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.526+0000 7f6b317fa640 1 -- 172.21.15.66:0/119342878 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4172795079 0 0) 0x7f6b340b06d0 con 0x7f6b340a98f0 2024-03-25T18:07:59.526 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.526+0000 7f6b317fa640 1 -- 172.21.15.66:0/119342878 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b18004eb0 con 0x7f6b340a98f0 2024-03-25T18:07:59.526 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.526+0000 7f6b317fa640 1 -- 172.21.15.66:0/119342878 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f6b280018d0 con 0x7f6b340a98f0 2024-03-25T18:07:59.526 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.526+0000 7f6b317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:59.526 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.526+0000 7f6b317fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:59.526 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.526+0000 7f6b317fa640 10 monclient(hunting): dump: 2024-03-25T18:07:59.527 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:59.527 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:59.527 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:59.527 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:59.527 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:59.527 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:59.527 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:59.527 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:59.527 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:59.527 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:59.536 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.534+0000 7f6b317fa640 1 -- 172.21.15.66:0/119342878 <== mon.2 v1:172.21.15.66:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 796410205 0 0) 0x7f6b340b03f0 con 0x7f6b341430c0 2024-03-25T18:07:59.536 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.535+0000 7f6b317fa640 1 -- 172.21.15.66:0/119342878 >> v1:172.21.15.193:6789/0 conn(0x7f6b340a98f0 legacy=0x7f6b341da7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:59.536 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.535+0000 7f6b317fa640 1 -- 172.21.15.66:0/119342878 >> v1:172.21.15.66:6789/0 conn(0x7f6b3413f710 legacy=0x7f6b340af080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:59.536 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.535+0000 7f6b317fa640 10 monclient: _finish_hunting 0 2024-03-25T18:07:59.536 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.535+0000 7f6b317fa640 1 monclient: found mon.c 2024-03-25T18:07:59.536 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.535+0000 7f6b317fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:59.536 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.535+0000 7f6b317fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:07:59.536 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.535+0000 7f6b317fa640 1 -- 172.21.15.66:0/119342878 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b341d7a10 con 0x7f6b341430c0 2024-03-25T18:07:59.536 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.535+0000 7f6b317fa640 10 monclient: _finish_auth 0 2024-03-25T18:07:59.536 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.535+0000 7f6b317fa640 10 monclient: _check_auth_tickets 2024-03-25T18:07:59.536 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.535+0000 7f6b317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:59.537 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.535+0000 7f6b3b1ba640 5 monclient: authenticate success, global_id 4409 2024-03-25T18:07:59.537 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.535+0000 7f6b3b1ba640 10 monclient: _renew_subs 2024-03-25T18:07:59.537 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.535+0000 7f6b3b1ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:07:59.537 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.535+0000 7f6b3b1ba640 1 -- 172.21.15.66:0/119342878 --> v1:172.21.15.66:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b341d7f30 con 0x7f6b341430c0 2024-03-25T18:07:59.537 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.535+0000 7f6b3b1ba640 10 client.4409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:59.537 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.535+0000 7f6b3b1ba640 10 monclient: _renew_subs 2024-03-25T18:07:59.537 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.535+0000 7f6b3b1ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:07:59.537 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.535+0000 7f6b3b1ba640 1 -- 172.21.15.66:0/119342878 --> v1:172.21.15.66:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b340b1040 con 0x7f6b341430c0 2024-03-25T18:07:59.537 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.535+0000 7f6b3b1ba640 1 librados: init done 2024-03-25T18:07:59.538 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.536+0000 7f6b3b1ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:59.538 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.536+0000 7f6b3b1ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:59.538 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.536+0000 7f6b3b1ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:07:59.538 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.536+0000 7f6b3b1ba640 1 -- 172.21.15.66:0/119342878 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b341d8230 con 0x7f6b341430c0 2024-03-25T18:07:59.865 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.865+0000 7f6b317fa640 1 -- 172.21.15.66:0/119342878 <== mon.2 v1:172.21.15.66:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b2c002020 con 0x7f6b341430c0 2024-03-25T18:07:59.865 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.865+0000 7f6b317fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:59.865 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.865+0000 7f6b317fa640 1 -- 172.21.15.66:0/119342878 <== mon.2 v1:172.21.15.66:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f6b2c004d80 con 0x7f6b341430c0 2024-03-25T18:07:59.865 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.865+0000 7f6b317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:59.866 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.865+0000 7f6b317fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:59.866 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.865+0000 7f6b317fa640 10 monclient: dump: 2024-03-25T18:07:59.866 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:07:59.866 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:07:59.866 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:59.866 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:07:59.866 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:59.866 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:07:59.866 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:07:59.866 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:07:59.866 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:07:59.867 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:07:59.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.865+0000 7f6b32ffd640 4 set_mon_vals no callback set 2024-03-25T18:07:59.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.865+0000 7f6b317fa640 1 -- 172.21.15.66:0/119342878 <== mon.2 v1:172.21.15.66:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f6b2c01ad90 con 0x7f6b341430c0 2024-03-25T18:07:59.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.866+0000 7f6b38f56640 10 monclient: get_auth_request con 0x7f6b1806ddf0 auth_method 0 2024-03-25T18:07:59.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.866+0000 7f6b317fa640 10 client.4409.objecter ms_handle_connect 0x7f6b1806ddf0 2024-03-25T18:07:59.872 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.872+0000 7f6b317fa640 1 -- 172.21.15.66:0/119342878 <== mon.2 v1:172.21.15.66:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f6b2c054ba0 con 0x7f6b341430c0 2024-03-25T18:07:59.872 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.872+0000 7f6b317fa640 10 client.4409.objecter ms_dispatch 0x7f6b34000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:07:59.872 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.872+0000 7f6b317fa640 3 client.4409.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:07:59.872 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.872+0000 7f6b317fa640 3 client.4409.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:07:59.872 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.872+0000 7f6b317fa640 20 client.4409.objecter dump_active .. 0 homeless 2024-03-25T18:07:59.873 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.872+0000 7f6b317fa640 1 -- 172.21.15.66:0/119342878 <== mon.2 v1:172.21.15.66:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6b2c0504e0 con 0x7f6b341430c0 2024-03-25T18:07:59.873 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.872+0000 7f6b317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:59.873 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:07:59.872+0000 7f6b317fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:00.134 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:00.133+0000 7f6b3b1ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:00.134 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:00.133+0000 7f6b3b1ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:00.134 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:00.133+0000 7f6b3b1ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:00.134 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:00.133+0000 7f6b3b1ba640 1 -- 172.21.15.66:0/119342878 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b34074cc0 con 0x7f6b341430c0 2024-03-25T18:08:00.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:00.157+0000 7f6b317fa640 1 -- 172.21.15.66:0/119342878 <== mon.2 v1:172.21.15.66:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f6b2c0525e0 con 0x7f6b341430c0 2024-03-25T18:08:00.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:00.157+0000 7f6b317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:00.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:00.157+0000 7f6b317fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:00.157 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:08:00.158 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:00.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:00.160+0000 7f6b3b1ba640 10 librados: watch_flush enter 2024-03-25T18:08:00.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:00.160+0000 7f6b3b1ba640 10 librados: watch_flush exit 2024-03-25T18:08:00.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:00.160+0000 7f6b3b1ba640 20 client.4409.objecter shutdown clearing up homeless session... 2024-03-25T18:08:00.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:00.160+0000 7f6b3b1ba640 10 client.4409.objecter successfully canceled tick 2024-03-25T18:08:00.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:00.160+0000 7f6b3b1ba640 1 -- 172.21.15.66:0/119342878 >> v1:172.21.15.193:6800/2275676803 conn(0x7f6b1806ddf0 legacy=0x7f6b180702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:00.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:00.160+0000 7f6b3b1ba640 10 monclient: shutdown 2024-03-25T18:08:00.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:00.160+0000 7f6b3b1ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:00.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:00.160+0000 7f6b3b1ba640 1 -- 172.21.15.66:0/119342878 >> v1:172.21.15.66:6790/0 conn(0x7f6b341430c0 legacy=0x7f6b341daf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:00.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:00.160+0000 7f6b3b1ba640 1 -- 172.21.15.66:0/119342878 shutdown_connections 2024-03-25T18:08:00.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:00.160+0000 7f6b3b1ba640 1 -- 172.21.15.66:0/119342878 >> 172.21.15.66:0/119342878 conn(0x7f6b340b7ea0 msgr2=0x7f6b34074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:00.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:00.160+0000 7f6b3b1ba640 1 -- 172.21.15.66:0/119342878 shutdown_connections 2024-03-25T18:08:00.162 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:00.160+0000 7f6b3b1ba640 1 -- 172.21.15.66:0/119342878 wait complete. 2024-03-25T18:08:00.162 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:00.160+0000 7f6b3b1ba640 1 librados: shutdown 2024-03-25T18:08:00.171 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:08:04.256 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:04:35.236 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:04.262 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:04:35.243 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:04.300 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:04:35.281 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:04.314 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:04:35.272 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:04.317 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:04:35.298 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:04.325 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:04:35.284 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:04.334 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:04:35.315 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:04.359 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:04:35.318 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:04.370 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:04:35.350 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:06.979 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:07.099 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.099+0000 7fec03df4640 10 monclient: get_monmap_and_config 2024-03-25T18:08:07.100 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.099+0000 7fec03df4640 10 monclient: build_initial_monmap 2024-03-25T18:08:07.100 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.099+0000 7fec03df4640 10 monclient: monmap: 2024-03-25T18:08:07.100 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:08:07.100 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:07.100 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:08:07.100088+0000 2024-03-25T18:08:07.100 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:08:07.100088+0000 2024-03-25T18:08:07.100 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:07.100 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:07.100 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:08:07.100 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:08:07.100 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:08:07.100 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:07.102 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.100+0000 7fec03df4640 1 Processor -- start 2024-03-25T18:08:07.102 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.100+0000 7fec03df4640 1 -- start start 2024-03-25T18:08:07.102 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.100+0000 7fec03df4640 10 monclient: init 2024-03-25T18:08:07.102 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.101+0000 7fec03df4640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:07.102 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.101+0000 7fec03df4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:08:07.102 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.101+0000 7fec03df4640 10 monclient(hunting): picked mon.noname-a con 0x7febfc0b4d60 addr v1:172.21.15.66:6789/0 2024-03-25T18:08:07.102 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.101+0000 7fec03df4640 10 monclient(hunting): picked mon.noname-b con 0x7febfc0b1730 addr v1:172.21.15.66:6790/0 2024-03-25T18:08:07.102 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.101+0000 7fec03df4640 10 monclient(hunting): picked mon.noname-c con 0x7febfc14be90 addr v1:172.21.15.193:6789/0 2024-03-25T18:08:07.102 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.101+0000 7fec03df4640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febfc0af940 con 0x7febfc0b4d60 2024-03-25T18:08:07.102 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.101+0000 7fec03df4640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febfc0b0a70 con 0x7febfc14be90 2024-03-25T18:08:07.102 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.101+0000 7fec03df4640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febfc0b0d50 con 0x7febfc0b1730 2024-03-25T18:08:07.103 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.101+0000 7fec03df4640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:07.103 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.101+0000 7fec03df4640 10 monclient(hunting): authenticate will time out at 7236.160813s 2024-03-25T18:08:07.111 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.110+0000 7fec01b90640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7febfc0b4d60 0x7febfc13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:49396/0 (socket says 172.21.15.66:49396) 2024-03-25T18:08:07.111 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.110+0000 7fec01b90640 1 -- 172.21.15.66:0/4136142837 learned_addr learned my addr 172.21.15.66:0/4136142837 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:08:07.126 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.126+0000 7fec00b8e640 1 -- 172.21.15.66:0/4136142837 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3360589175 0 0) 0x7febfc0b0a70 con 0x7febfc14be90 2024-03-25T18:08:07.126 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.126+0000 7fec00b8e640 10 monclient(hunting): _negotiate 2024-03-25T18:08:07.126 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.126+0000 7fec00b8e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:07.126 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.126+0000 7fec00b8e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:07.127 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.126+0000 7fec00b8e640 10 monclient(hunting): my global_id is 4408 2024-03-25T18:08:07.127 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.126+0000 7fec00b8e640 1 -- 172.21.15.66:0/4136142837 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febe8003520 con 0x7febfc14be90 2024-03-25T18:08:07.133 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.132+0000 7fec00b8e640 1 -- 172.21.15.66:0/4136142837 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2746856473 0 0) 0x7febfc0af940 con 0x7febfc0b4d60 2024-03-25T18:08:07.133 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.132+0000 7fec00b8e640 10 monclient(hunting): _negotiate 2024-03-25T18:08:07.133 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.132+0000 7fec00b8e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:07.133 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.132+0000 7fec00b8e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:07.133 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.132+0000 7fec00b8e640 10 monclient(hunting): my global_id is 4410 2024-03-25T18:08:07.133 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.132+0000 7fec00b8e640 1 -- 172.21.15.66:0/4136142837 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febfc0b0a70 con 0x7febfc0b4d60 2024-03-25T18:08:07.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.136+0000 7fec00b8e640 1 -- 172.21.15.66:0/4136142837 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2809108381 0 0) 0x7febe8003520 con 0x7febfc14be90 2024-03-25T18:08:07.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.136+0000 7fec00b8e640 1 -- 172.21.15.66:0/4136142837 >> v1:172.21.15.66:6790/0 conn(0x7febfc0b1730 legacy=0x7febfc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:07.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.136+0000 7fec00b8e640 1 -- 172.21.15.66:0/4136142837 >> v1:172.21.15.66:6789/0 conn(0x7febfc0b4d60 legacy=0x7febfc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:07.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.136+0000 7fec00b8e640 10 monclient: _finish_hunting 0 2024-03-25T18:08:07.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.136+0000 7fec00b8e640 1 monclient: found mon.noname-c 2024-03-25T18:08:07.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.136+0000 7fec00b8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:07.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.136+0000 7fec00b8e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:08:07.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.136+0000 7fec00b8e640 1 -- 172.21.15.66:0/4136142837 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febfc0b1030 con 0x7febfc14be90 2024-03-25T18:08:07.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.136+0000 7fec00b8e640 10 monclient: _finish_auth 0 2024-03-25T18:08:07.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.136+0000 7fec00b8e640 10 monclient: _check_auth_tickets 2024-03-25T18:08:07.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.136+0000 7fec00b8e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:07.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.136+0000 7fec00b8e640 1 -- 172.21.15.66:0/4136142837 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7febf4001a10 con 0x7febfc14be90 2024-03-25T18:08:07.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.136+0000 7fec00b8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:07.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.136+0000 7fec00b8e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:07.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.136+0000 7fec00b8e640 10 monclient: dump: 2024-03-25T18:08:07.138 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:07.138 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:07.138 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:07.138 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:07.138 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:07.138 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:07.138 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:07.139 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:07.139 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:07.139 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:07.139 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.136+0000 7fec03df4640 5 monclient: authenticate success, global_id 4408 2024-03-25T18:08:07.139 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.136+0000 7fec03df4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:07.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.144+0000 7fec00b8e640 1 -- 172.21.15.66:0/4136142837 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febf4001b70 con 0x7febfc14be90 2024-03-25T18:08:07.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.144+0000 7fec00b8e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:07.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.144+0000 7fec00b8e640 1 -- 172.21.15.66:0/4136142837 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7febf4001f80 con 0x7febfc14be90 2024-03-25T18:08:07.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.144+0000 7fec00b8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:07.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.144+0000 7fec00b8e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:07.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.144+0000 7fec00b8e640 10 monclient: dump: 2024-03-25T18:08:07.145 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:07.145 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:07.145 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:07.145 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:07.145 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:07.145 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:07.145 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:07.145 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:07.145 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:07.145 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:07.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.144+0000 7fec03df4640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:07.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.144+0000 7fec03df4640 4 set_mon_vals no callback set 2024-03-25T18:08:07.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.144+0000 7fec03df4640 10 monclient: shutdown 2024-03-25T18:08:07.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.144+0000 7fec03df4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:07.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.144+0000 7fec03df4640 1 -- 172.21.15.66:0/4136142837 >> v1:172.21.15.193:6789/0 conn(0x7febfc14be90 legacy=0x7febfc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:07.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.145+0000 7fec03df4640 1 -- 172.21.15.66:0/4136142837 shutdown_connections 2024-03-25T18:08:07.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.145+0000 7fec03df4640 1 -- 172.21.15.66:0/4136142837 >> 172.21.15.66:0/4136142837 conn(0x7febfc0617a0 msgr2=0x7febfc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:07.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.145+0000 7fec03df4640 1 -- 172.21.15.66:0/4136142837 shutdown_connections 2024-03-25T18:08:07.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.145+0000 7fec03df4640 1 -- 172.21.15.66:0/4136142837 wait complete. 2024-03-25T18:08:07.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.145+0000 7fec03df4640 10 monclient: build_initial_monmap 2024-03-25T18:08:07.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.145+0000 7fec03df4640 10 monclient: monmap: 2024-03-25T18:08:07.148 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:08:07.148 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:07.148 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:08:07.148 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:08:07.148 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:07.148 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:08:07.149 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:08:07.149 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:08:07.149 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:08:07.149 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:07.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.146+0000 7fec03df4640 1 librados: starting msgr at 2024-03-25T18:08:07.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.146+0000 7fec03df4640 1 librados: starting objecter 2024-03-25T18:08:07.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.146+0000 7fec03df4640 1 Processor -- start 2024-03-25T18:08:07.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.146+0000 7fec03df4640 1 -- start start 2024-03-25T18:08:07.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.146+0000 7fec03df4640 1 librados: setting wanted keys 2024-03-25T18:08:07.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.146+0000 7fec03df4640 1 librados: calling monclient init 2024-03-25T18:08:07.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.146+0000 7fec03df4640 10 monclient: init 2024-03-25T18:08:07.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.147+0000 7fec03df4640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:07.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.147+0000 7fec03df4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:08:07.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.147+0000 7fec03df4640 10 monclient(hunting): picked mon.noname-a con 0x7febfc0b1730 addr v1:172.21.15.66:6789/0 2024-03-25T18:08:07.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.147+0000 7fec03df4640 10 monclient(hunting): picked mon.noname-b con 0x7febfc0b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T18:08:07.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.147+0000 7fec03df4640 10 monclient(hunting): picked mon.noname-c con 0x7febfc14be90 addr v1:172.21.15.66:6790/0 2024-03-25T18:08:07.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.147+0000 7fec03df4640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febfc1363f0 con 0x7febfc0b1730 2024-03-25T18:08:07.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.147+0000 7fec03df4640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febfc1374e0 con 0x7febfc0b4d60 2024-03-25T18:08:07.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.147+0000 7fec03df4640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febfc138140 con 0x7febfc14be90 2024-03-25T18:08:07.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.147+0000 7fec03df4640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:07.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.147+0000 7fec03df4640 10 monclient(hunting): authenticate will time out at 7236.206775s 2024-03-25T18:08:07.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.155+0000 7fec0138f640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7febfc0b4d60 0x7febfc12fd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:41062/0 (socket says 172.21.15.66:41062) 2024-03-25T18:08:07.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.155+0000 7fec0138f640 1 -- 172.21.15.66:0/3043876170 learned_addr learned my addr 172.21.15.66:0/3043876170 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:08:07.158 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.158+0000 7febf27fc640 10 client.?.objecter ms_handle_connect 0x7febfc0b4d60 2024-03-25T18:08:07.158 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.158+0000 7febf27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:07.162 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.162+0000 7febf27fc640 10 client.?.objecter ms_handle_connect 0x7febfc14be90 2024-03-25T18:08:07.162 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.162+0000 7febf27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:07.168 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.168+0000 7febf27fc640 10 client.?.objecter ms_handle_connect 0x7febfc0b1730 2024-03-25T18:08:07.168 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.168+0000 7febf27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:07.172 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.172+0000 7febf27fc640 1 -- 172.21.15.66:0/3043876170 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 350150298 0 0) 0x7febfc138140 con 0x7febfc14be90 2024-03-25T18:08:07.172 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.172+0000 7febf27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:07.172 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.172+0000 7febf27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:07.173 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.172+0000 7febf27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:07.173 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.172+0000 7febf27fc640 10 monclient(hunting): my global_id is 4415 2024-03-25T18:08:07.173 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.172+0000 7febf27fc640 1 -- 172.21.15.66:0/3043876170 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febe0004eb0 con 0x7febfc14be90 2024-03-25T18:08:07.180 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.180+0000 7febf27fc640 1 -- 172.21.15.66:0/3043876170 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200051611 0 0) 0x7febfc1374e0 con 0x7febfc0b4d60 2024-03-25T18:08:07.181 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.180+0000 7febf27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:07.181 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.180+0000 7febf27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:07.181 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.180+0000 7febf27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:07.181 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.180+0000 7febf27fc640 10 monclient(hunting): my global_id is 4411 2024-03-25T18:08:07.181 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.180+0000 7febf27fc640 1 -- 172.21.15.66:0/3043876170 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febfc138140 con 0x7febfc0b4d60 2024-03-25T18:08:07.182 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.182+0000 7febf27fc640 1 -- 172.21.15.66:0/3043876170 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 513668145 0 0) 0x7febe0004eb0 con 0x7febfc14be90 2024-03-25T18:08:07.183 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.182+0000 7febf27fc640 1 -- 172.21.15.66:0/3043876170 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febfc1374e0 con 0x7febfc14be90 2024-03-25T18:08:07.183 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.182+0000 7febf27fc640 1 -- 172.21.15.66:0/3043876170 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7febf4001ec0 con 0x7febfc14be90 2024-03-25T18:08:07.183 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.182+0000 7febf27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:07.183 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.182+0000 7febf27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:07.183 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.182+0000 7febf27fc640 10 monclient(hunting): dump: 2024-03-25T18:08:07.183 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:07.183 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:07.183 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:07.183 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:07.183 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:07.183 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:07.183 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:07.183 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:07.183 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:07.183 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:07.183 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.182+0000 7febf27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:07.190 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.189+0000 7febf27fc640 1 -- 172.21.15.66:0/3043876170 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 993854612 0 0) 0x7febfc138140 con 0x7febfc0b4d60 2024-03-25T18:08:07.190 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.189+0000 7febf27fc640 1 -- 172.21.15.66:0/3043876170 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febe0004eb0 con 0x7febfc0b4d60 2024-03-25T18:08:07.190 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.189+0000 7febf27fc640 1 -- 172.21.15.66:0/3043876170 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7febec001970 con 0x7febfc0b4d60 2024-03-25T18:08:07.190 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.189+0000 7febf27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:07.190 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.189+0000 7febf27fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:07.190 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.189+0000 7febf27fc640 10 monclient(hunting): dump: 2024-03-25T18:08:07.190 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:07.190 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:07.190 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:07.190 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:07.191 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:07.191 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:07.191 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:07.191 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:07.191 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:07.191 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:07.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.191+0000 7febf27fc640 1 -- 172.21.15.66:0/3043876170 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 348386119 0 0) 0x7febfc1363f0 con 0x7febfc0b1730 2024-03-25T18:08:07.192 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.191+0000 7febf27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:07.192 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.191+0000 7febf27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:07.192 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.191+0000 7febf27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:07.192 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.191+0000 7febf27fc640 10 monclient(hunting): my global_id is 4413 2024-03-25T18:08:07.192 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.191+0000 7febf27fc640 1 -- 172.21.15.66:0/3043876170 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febfc138140 con 0x7febfc0b1730 2024-03-25T18:08:07.203 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.203+0000 7febf27fc640 1 -- 172.21.15.66:0/3043876170 <== mon.2 v1:172.21.15.66:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3389321952 0 0) 0x7febfc1374e0 con 0x7febfc14be90 2024-03-25T18:08:07.204 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.203+0000 7febf27fc640 1 -- 172.21.15.66:0/3043876170 >> v1:172.21.15.193:6789/0 conn(0x7febfc0b4d60 legacy=0x7febfc12fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:07.204 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.203+0000 7febf27fc640 1 -- 172.21.15.66:0/3043876170 >> v1:172.21.15.66:6789/0 conn(0x7febfc0b1730 legacy=0x7febfc136dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:07.204 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.203+0000 7febf27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:08:07.204 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.203+0000 7febf27fc640 1 monclient: found mon.c 2024-03-25T18:08:07.204 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.203+0000 7febf27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:07.204 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.203+0000 7febf27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:07.204 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.203+0000 7febf27fc640 1 -- 172.21.15.66:0/3043876170 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febfc139090 con 0x7febfc14be90 2024-03-25T18:08:07.204 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.203+0000 7febf27fc640 10 monclient: _finish_auth 0 2024-03-25T18:08:07.204 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.203+0000 7febf27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:08:07.204 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.203+0000 7febf27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:07.204 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.203+0000 7fec03df4640 5 monclient: authenticate success, global_id 4415 2024-03-25T18:08:07.204 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.203+0000 7fec03df4640 10 monclient: _renew_subs 2024-03-25T18:08:07.204 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.203+0000 7fec03df4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:07.205 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.203+0000 7fec03df4640 1 -- 172.21.15.66:0/3043876170 --> v1:172.21.15.66:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febfc136030 con 0x7febfc14be90 2024-03-25T18:08:07.205 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.203+0000 7fec03df4640 10 client.4415.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:07.205 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.203+0000 7fec03df4640 10 monclient: _renew_subs 2024-03-25T18:08:07.205 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.203+0000 7fec03df4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:07.205 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.203+0000 7fec03df4640 1 -- 172.21.15.66:0/3043876170 --> v1:172.21.15.66:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7febfc130650 con 0x7febfc14be90 2024-03-25T18:08:07.205 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.203+0000 7fec03df4640 1 librados: init done 2024-03-25T18:08:07.205 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.204+0000 7fec03df4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:07.205 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.204+0000 7fec03df4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:07.205 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.204+0000 7fec03df4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:07.206 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.204+0000 7fec03df4640 1 -- 172.21.15.66:0/3043876170 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febfc0ae6c0 con 0x7febfc14be90 2024-03-25T18:08:07.426 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.426+0000 7febf27fc640 1 -- 172.21.15.66:0/3043876170 <== mon.2 v1:172.21.15.66:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febf40020e0 con 0x7febfc14be90 2024-03-25T18:08:07.426 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.426+0000 7febf27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:07.426 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.426+0000 7febf27fc640 1 -- 172.21.15.66:0/3043876170 <== mon.2 v1:172.21.15.66:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7febf4003dc0 con 0x7febfc14be90 2024-03-25T18:08:07.426 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.426+0000 7febf27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:07.427 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.426+0000 7febf27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:07.427 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.426+0000 7febf27fc640 10 monclient: dump: 2024-03-25T18:08:07.427 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:07.427 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:07.427 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:07.427 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:07.427 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:07.427 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:07.427 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:07.427 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:07.427 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:07.427 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:07.427 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.426+0000 7febf3fff640 4 set_mon_vals no callback set 2024-03-25T18:08:07.523 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.523+0000 7febf27fc640 1 -- 172.21.15.66:0/3043876170 <== mon.2 v1:172.21.15.66:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7febf4019d90 con 0x7febfc14be90 2024-03-25T18:08:07.524 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.524+0000 7fec01b90640 10 monclient: get_auth_request con 0x7febe006df50 auth_method 0 2024-03-25T18:08:07.524 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.524+0000 7febf27fc640 10 client.4415.objecter ms_handle_connect 0x7febe006df50 2024-03-25T18:08:07.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.530+0000 7febf27fc640 1 -- 172.21.15.66:0/3043876170 <== mon.2 v1:172.21.15.66:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7febf40538b0 con 0x7febfc14be90 2024-03-25T18:08:07.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.530+0000 7febf27fc640 10 client.4415.objecter ms_dispatch 0x7febfc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:07.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.530+0000 7febf27fc640 3 client.4415.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:07.531 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.530+0000 7febf27fc640 3 client.4415.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:07.531 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.530+0000 7febf27fc640 20 client.4415.objecter dump_active .. 0 homeless 2024-03-25T18:08:07.531 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.530+0000 7febf27fc640 1 -- 172.21.15.66:0/3043876170 <== mon.2 v1:172.21.15.66:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7febf404f1f0 con 0x7febfc14be90 2024-03-25T18:08:07.531 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.530+0000 7febf27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:07.531 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.530+0000 7febf27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:07.772 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.771+0000 7fec03df4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:07.772 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.771+0000 7fec03df4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:07.772 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.771+0000 7fec03df4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:07.772 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.771+0000 7fec03df4640 1 -- 172.21.15.66:0/3043876170 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febfc0627b0 con 0x7febfc14be90 2024-03-25T18:08:07.796 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.795+0000 7febf27fc640 1 -- 172.21.15.66:0/3043876170 <== mon.2 v1:172.21.15.66:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7febf40512f0 con 0x7febfc14be90 2024-03-25T18:08:07.796 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.795+0000 7febf27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:07.796 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.795+0000 7febf27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:07.796 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:08:07.796 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:07.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.798+0000 7fec03df4640 10 librados: watch_flush enter 2024-03-25T18:08:07.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.798+0000 7fec03df4640 10 librados: watch_flush exit 2024-03-25T18:08:07.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.798+0000 7fec03df4640 20 client.4415.objecter shutdown clearing up homeless session... 2024-03-25T18:08:07.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.798+0000 7fec03df4640 10 client.4415.objecter successfully canceled tick 2024-03-25T18:08:07.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.798+0000 7fec03df4640 1 -- 172.21.15.66:0/3043876170 >> v1:172.21.15.193:6800/2275676803 conn(0x7febe006df50 legacy=0x7febe0070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:07.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.798+0000 7fec03df4640 10 monclient: shutdown 2024-03-25T18:08:07.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.798+0000 7fec03df4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:07.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.798+0000 7fec03df4640 1 -- 172.21.15.66:0/3043876170 >> v1:172.21.15.66:6790/0 conn(0x7febfc14be90 legacy=0x7febfc143c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:07.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.798+0000 7fec03df4640 1 -- 172.21.15.66:0/3043876170 shutdown_connections 2024-03-25T18:08:07.800 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.798+0000 7fec03df4640 1 -- 172.21.15.66:0/3043876170 >> 172.21.15.66:0/3043876170 conn(0x7febfc0617a0 msgr2=0x7febfc13c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:07.800 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.798+0000 7fec03df4640 1 -- 172.21.15.66:0/3043876170 shutdown_connections 2024-03-25T18:08:07.800 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.798+0000 7fec03df4640 1 -- 172.21.15.66:0/3043876170 wait complete. 2024-03-25T18:08:07.800 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:07.798+0000 7fec03df4640 1 librados: shutdown 2024-03-25T18:08:07.810 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:08:14.617 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:14.737 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.736+0000 7fa96eeed640 10 monclient: get_monmap_and_config 2024-03-25T18:08:14.737 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.736+0000 7fa96eeed640 10 monclient: build_initial_monmap 2024-03-25T18:08:14.737 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.736+0000 7fa96eeed640 10 monclient: monmap: 2024-03-25T18:08:14.737 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:08:14.738 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:14.738 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:08:14.737459+0000 2024-03-25T18:08:14.738 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:08:14.737459+0000 2024-03-25T18:08:14.738 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:14.738 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:14.738 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:08:14.738 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:08:14.738 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:08:14.738 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:14.739 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.737+0000 7fa96eeed640 1 Processor -- start 2024-03-25T18:08:14.739 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.738+0000 7fa96eeed640 1 -- start start 2024-03-25T18:08:14.739 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.738+0000 7fa96eeed640 10 monclient: init 2024-03-25T18:08:14.739 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.738+0000 7fa96eeed640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:14.740 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.738+0000 7fa96eeed640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:08:14.740 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.738+0000 7fa96eeed640 10 monclient(hunting): picked mon.noname-b con 0x7fa9680a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:08:14.740 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.738+0000 7fa96eeed640 10 monclient(hunting): picked mon.noname-c con 0x7fa96813f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:08:14.740 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.738+0000 7fa96eeed640 10 monclient(hunting): picked mon.noname-a con 0x7fa9681430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:08:14.740 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.738+0000 7fa96eeed640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa968142520 con 0x7fa9681430c0 2024-03-25T18:08:14.740 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.738+0000 7fa96eeed640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9680b28a0 con 0x7fa96813f710 2024-03-25T18:08:14.740 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.738+0000 7fa96eeed640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9680b2a30 con 0x7fa9680a98f0 2024-03-25T18:08:14.740 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.738+0000 7fa96eeed640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:14.740 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.738+0000 7fa96eeed640 10 monclient(hunting): authenticate will time out at 7243.798011s 2024-03-25T18:08:14.745 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.744+0000 7fa967fff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fa96813f710 0x7fa9681429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:38564/0 (socket says 172.21.15.66:38564) 2024-03-25T18:08:14.745 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.744+0000 7fa967fff640 1 -- 172.21.15.66:0/3445836635 learned_addr learned my addr 172.21.15.66:0/3445836635 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:08:14.757 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.757+0000 7fa9677fe640 1 -- 172.21.15.66:0/3445836635 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846985967 0 0) 0x7fa9680b28a0 con 0x7fa96813f710 2024-03-25T18:08:14.757 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.757+0000 7fa9677fe640 10 monclient(hunting): _negotiate 2024-03-25T18:08:14.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.757+0000 7fa9677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:14.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.757+0000 7fa9677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:14.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.757+0000 7fa9677fe640 10 monclient(hunting): my global_id is 4414 2024-03-25T18:08:14.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.757+0000 7fa9677fe640 1 -- 172.21.15.66:0/3445836635 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa958003520 con 0x7fa96813f710 2024-03-25T18:08:14.761 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.760+0000 7fa9677fe640 1 -- 172.21.15.66:0/3445836635 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202819274 0 0) 0x7fa9680b2a30 con 0x7fa9680a98f0 2024-03-25T18:08:14.761 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.760+0000 7fa9677fe640 10 monclient(hunting): _negotiate 2024-03-25T18:08:14.761 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.760+0000 7fa9677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:14.761 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.760+0000 7fa9677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:14.761 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.760+0000 7fa9677fe640 10 monclient(hunting): my global_id is 4418 2024-03-25T18:08:14.761 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.760+0000 7fa9677fe640 1 -- 172.21.15.66:0/3445836635 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9680b28a0 con 0x7fa9680a98f0 2024-03-25T18:08:14.766 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.765+0000 7fa9677fe640 1 -- 172.21.15.66:0/3445836635 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1449482507 0 0) 0x7fa958003520 con 0x7fa96813f710 2024-03-25T18:08:14.766 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.766+0000 7fa9677fe640 1 -- 172.21.15.66:0/3445836635 >> v1:172.21.15.66:6790/0 conn(0x7fa9680a98f0 legacy=0x7fa9680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:14.766 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.766+0000 7fa9677fe640 1 -- 172.21.15.66:0/3445836635 >> v1:172.21.15.66:6789/0 conn(0x7fa9681430c0 legacy=0x7fa96814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:14.767 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.766+0000 7fa9677fe640 10 monclient: _finish_hunting 0 2024-03-25T18:08:14.767 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.766+0000 7fa9677fe640 1 monclient: found mon.noname-c 2024-03-25T18:08:14.767 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.766+0000 7fa9677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:14.767 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.766+0000 7fa9677fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:08:14.767 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.766+0000 7fa9677fe640 1 -- 172.21.15.66:0/3445836635 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9680b44c0 con 0x7fa96813f710 2024-03-25T18:08:14.767 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.766+0000 7fa9677fe640 10 monclient: _finish_auth 0 2024-03-25T18:08:14.767 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.766+0000 7fa9677fe640 10 monclient: _check_auth_tickets 2024-03-25T18:08:14.767 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.766+0000 7fa9677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:14.767 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.766+0000 7fa9677fe640 1 -- 172.21.15.66:0/3445836635 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fa954001690 con 0x7fa96813f710 2024-03-25T18:08:14.767 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.766+0000 7fa9677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:14.767 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.766+0000 7fa9677fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:14.767 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.766+0000 7fa9677fe640 10 monclient: dump: 2024-03-25T18:08:14.767 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:14.767 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:14.768 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:14.768 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:14.768 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:14.768 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:14.768 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:14.768 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:14.768 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:14.768 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:14.768 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.766+0000 7fa96eeed640 5 monclient: authenticate success, global_id 4414 2024-03-25T18:08:14.768 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.766+0000 7fa96eeed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:14.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.790+0000 7fa9677fe640 1 -- 172.21.15.66:0/3445836635 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9540017f0 con 0x7fa96813f710 2024-03-25T18:08:14.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.790+0000 7fa9677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:14.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.790+0000 7fa9677fe640 1 -- 172.21.15.66:0/3445836635 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fa954001c00 con 0x7fa96813f710 2024-03-25T18:08:14.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.790+0000 7fa9677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:14.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.790+0000 7fa9677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:14.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.790+0000 7fa9677fe640 10 monclient: dump: 2024-03-25T18:08:14.791 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:14.791 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:14.792 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:14.792 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:14.792 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:14.792 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:14.792 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:14.792 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:14.792 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:14.792 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:14.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.790+0000 7fa96eeed640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:14.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.790+0000 7fa96eeed640 4 set_mon_vals no callback set 2024-03-25T18:08:14.793 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.791+0000 7fa96eeed640 10 monclient: shutdown 2024-03-25T18:08:14.793 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.791+0000 7fa96eeed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:14.793 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.791+0000 7fa96eeed640 1 -- 172.21.15.66:0/3445836635 >> v1:172.21.15.193:6789/0 conn(0x7fa96813f710 legacy=0x7fa9681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:14.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.792+0000 7fa96eeed640 1 -- 172.21.15.66:0/3445836635 shutdown_connections 2024-03-25T18:08:14.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.792+0000 7fa96eeed640 1 -- 172.21.15.66:0/3445836635 >> 172.21.15.66:0/3445836635 conn(0x7fa9680b7ea0 msgr2=0x7fa9680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:14.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.792+0000 7fa96eeed640 1 -- 172.21.15.66:0/3445836635 shutdown_connections 2024-03-25T18:08:14.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.792+0000 7fa96eeed640 1 -- 172.21.15.66:0/3445836635 wait complete. 2024-03-25T18:08:14.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.793+0000 7fa96eeed640 10 monclient: build_initial_monmap 2024-03-25T18:08:14.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.793+0000 7fa96eeed640 10 monclient: monmap: 2024-03-25T18:08:14.794 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:08:14.794 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:14.794 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:08:14.794 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:08:14.794 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:14.794 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:08:14.794 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:08:14.794 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:08:14.795 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:08:14.795 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:14.796 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.793+0000 7fa96eeed640 1 librados: starting msgr at 2024-03-25T18:08:14.796 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.793+0000 7fa96eeed640 1 librados: starting objecter 2024-03-25T18:08:14.796 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.794+0000 7fa96eeed640 1 Processor -- start 2024-03-25T18:08:14.796 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.794+0000 7fa96eeed640 1 -- start start 2024-03-25T18:08:14.796 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.794+0000 7fa96eeed640 1 librados: setting wanted keys 2024-03-25T18:08:14.796 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.794+0000 7fa96eeed640 1 librados: calling monclient init 2024-03-25T18:08:14.796 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.794+0000 7fa96eeed640 10 monclient: init 2024-03-25T18:08:14.796 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.794+0000 7fa96eeed640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:14.796 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.794+0000 7fa96eeed640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:08:14.796 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.794+0000 7fa96eeed640 10 monclient(hunting): picked mon.noname-b con 0x7fa9680a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:08:14.797 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.794+0000 7fa96eeed640 10 monclient(hunting): picked mon.noname-c con 0x7fa96813f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:08:14.797 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.794+0000 7fa96eeed640 10 monclient(hunting): picked mon.noname-a con 0x7fa9681430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:08:14.797 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.794+0000 7fa96eeed640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9681d77a0 con 0x7fa9681430c0 2024-03-25T18:08:14.797 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.794+0000 7fa96eeed640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa968138e40 con 0x7fa9680a98f0 2024-03-25T18:08:14.797 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.794+0000 7fa96eeed640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9680aba30 con 0x7fa96813f710 2024-03-25T18:08:14.797 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.794+0000 7fa96eeed640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:14.797 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.794+0000 7fa96eeed640 10 monclient(hunting): authenticate will time out at 7243.853998s 2024-03-25T18:08:14.809 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.809+0000 7fa967fff640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7fa96813f710 0x7fa9681d7c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:53684/0 (socket says 172.21.15.66:53684) 2024-03-25T18:08:14.809 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.809+0000 7fa967fff640 1 -- 172.21.15.66:0/3192329527 learned_addr learned my addr 172.21.15.66:0/3192329527 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:08:14.810 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.810+0000 7fa9657fa640 10 client.?.objecter ms_handle_connect 0x7fa96813f710 2024-03-25T18:08:14.810 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.810+0000 7fa9657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:14.813 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.813+0000 7fa9657fa640 1 -- 172.21.15.66:0/3192329527 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849435296 0 0) 0x7fa9680aba30 con 0x7fa96813f710 2024-03-25T18:08:14.813 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.813+0000 7fa9657fa640 10 monclient(hunting): _negotiate 2024-03-25T18:08:14.814 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.813+0000 7fa9657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:14.814 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.813+0000 7fa9657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:14.814 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.813+0000 7fa9657fa640 10 monclient(hunting): my global_id is 4421 2024-03-25T18:08:14.814 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.813+0000 7fa9657fa640 1 -- 172.21.15.66:0/3192329527 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa94c004eb0 con 0x7fa96813f710 2024-03-25T18:08:14.814 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.814+0000 7fa9657fa640 10 client.?.objecter ms_handle_connect 0x7fa9681430c0 2024-03-25T18:08:14.814 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.814+0000 7fa9657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:14.818 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.817+0000 7fa9657fa640 1 -- 172.21.15.66:0/3192329527 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 608726467 0 0) 0x7fa9681d77a0 con 0x7fa9681430c0 2024-03-25T18:08:14.818 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.817+0000 7fa9657fa640 10 monclient(hunting): _negotiate 2024-03-25T18:08:14.818 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.817+0000 7fa9657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:14.818 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.817+0000 7fa9657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:14.818 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.817+0000 7fa9657fa640 10 monclient(hunting): my global_id is 4419 2024-03-25T18:08:14.818 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.817+0000 7fa9657fa640 1 -- 172.21.15.66:0/3192329527 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9680aba30 con 0x7fa9681430c0 2024-03-25T18:08:14.822 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.821+0000 7fa9657fa640 1 -- 172.21.15.66:0/3192329527 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1450605655 0 0) 0x7fa94c004eb0 con 0x7fa96813f710 2024-03-25T18:08:14.822 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.821+0000 7fa9657fa640 1 -- 172.21.15.66:0/3192329527 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9681d77a0 con 0x7fa96813f710 2024-03-25T18:08:14.822 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.821+0000 7fa9657fa640 1 -- 172.21.15.66:0/3192329527 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fa954001c40 con 0x7fa96813f710 2024-03-25T18:08:14.822 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.821+0000 7fa9657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:14.822 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.821+0000 7fa9657fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:14.822 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.821+0000 7fa9657fa640 10 monclient(hunting): dump: 2024-03-25T18:08:14.822 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:14.822 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:14.823 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:14.823 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:14.823 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:14.823 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:14.823 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:14.823 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:14.823 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:14.823 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:14.823 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.821+0000 7fa9657fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:14.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.828+0000 7fa9657fa640 10 client.?.objecter ms_handle_connect 0x7fa9680a98f0 2024-03-25T18:08:14.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.828+0000 7fa9657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:14.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.828+0000 7fa9657fa640 1 -- 172.21.15.66:0/3192329527 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 65688951 0 0) 0x7fa9680aba30 con 0x7fa9681430c0 2024-03-25T18:08:14.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.828+0000 7fa9657fa640 1 -- 172.21.15.66:0/3192329527 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa94c004eb0 con 0x7fa9681430c0 2024-03-25T18:08:14.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.828+0000 7fa9657fa640 1 -- 172.21.15.66:0/3192329527 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fa960001b00 con 0x7fa9681430c0 2024-03-25T18:08:14.829 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.828+0000 7fa9657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:14.829 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.828+0000 7fa9657fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:14.829 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.828+0000 7fa9657fa640 10 monclient(hunting): dump: 2024-03-25T18:08:14.829 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:14.829 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:14.829 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:14.829 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:14.829 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:14.829 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:14.829 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:14.829 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:14.829 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:14.829 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:14.833 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.832+0000 7fa9657fa640 1 -- 172.21.15.66:0/3192329527 <== mon.2 v1:172.21.15.66:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4214828103 0 0) 0x7fa9681d77a0 con 0x7fa96813f710 2024-03-25T18:08:14.833 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.833+0000 7fa9657fa640 1 -- 172.21.15.66:0/3192329527 >> v1:172.21.15.193:6789/0 conn(0x7fa9680a98f0 legacy=0x7fa968138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:14.834 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.833+0000 7fa9657fa640 1 -- 172.21.15.66:0/3192329527 >> v1:172.21.15.66:6789/0 conn(0x7fa9681430c0 legacy=0x7fa968134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:14.834 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.833+0000 7fa9657fa640 10 monclient: _finish_hunting 0 2024-03-25T18:08:14.834 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.833+0000 7fa9657fa640 1 monclient: found mon.c 2024-03-25T18:08:14.834 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.833+0000 7fa9657fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:14.834 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.833+0000 7fa9657fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:14.834 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.833+0000 7fa9657fa640 1 -- 172.21.15.66:0/3192329527 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9681d8340 con 0x7fa96813f710 2024-03-25T18:08:14.834 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.833+0000 7fa9657fa640 10 monclient: _finish_auth 0 2024-03-25T18:08:14.834 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.833+0000 7fa9657fa640 10 monclient: _check_auth_tickets 2024-03-25T18:08:14.834 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.833+0000 7fa9657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:14.834 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.833+0000 7fa96eeed640 5 monclient: authenticate success, global_id 4421 2024-03-25T18:08:14.834 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.833+0000 7fa96eeed640 10 monclient: _renew_subs 2024-03-25T18:08:14.834 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.833+0000 7fa96eeed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:14.834 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.833+0000 7fa96eeed640 1 -- 172.21.15.66:0/3192329527 --> v1:172.21.15.66:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa968134820 con 0x7fa96813f710 2024-03-25T18:08:14.834 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.833+0000 7fa96eeed640 10 client.4421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:14.835 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.833+0000 7fa96eeed640 10 monclient: _renew_subs 2024-03-25T18:08:14.835 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.833+0000 7fa96eeed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:14.835 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.833+0000 7fa96eeed640 1 -- 172.21.15.66:0/3192329527 --> v1:172.21.15.66:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9681380c0 con 0x7fa96813f710 2024-03-25T18:08:14.835 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.833+0000 7fa96eeed640 1 librados: init done 2024-03-25T18:08:14.835 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.834+0000 7fa96eeed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:14.835 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.834+0000 7fa96eeed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:14.835 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.834+0000 7fa96eeed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:14.836 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:14.834+0000 7fa96eeed640 1 -- 172.21.15.66:0/3192329527 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9680627b0 con 0x7fa96813f710 2024-03-25T18:08:15.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.155+0000 7fa9657fa640 1 -- 172.21.15.66:0/3192329527 <== mon.2 v1:172.21.15.66:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa954004a90 con 0x7fa96813f710 2024-03-25T18:08:15.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.155+0000 7fa9657fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:15.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.155+0000 7fa9657fa640 1 -- 172.21.15.66:0/3192329527 <== mon.2 v1:172.21.15.66:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fa954004de0 con 0x7fa96813f710 2024-03-25T18:08:15.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.155+0000 7fa9657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:15.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.155+0000 7fa9657fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:15.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.155+0000 7fa9657fa640 10 monclient: dump: 2024-03-25T18:08:15.156 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:15.156 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:15.156 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:15.157 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:15.157 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:15.157 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:15.157 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:15.157 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:15.157 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:15.157 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:15.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.155+0000 7fa966ffd640 4 set_mon_vals no callback set 2024-03-25T18:08:15.158 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.156+0000 7fa9657fa640 1 -- 172.21.15.66:0/3192329527 <== mon.2 v1:172.21.15.66:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7fa954005000 con 0x7fa96813f710 2024-03-25T18:08:15.158 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.157+0000 7fa96cc89640 10 monclient: get_auth_request con 0x7fa94c06de40 auth_method 0 2024-03-25T18:08:15.158 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.157+0000 7fa9657fa640 10 client.4421.objecter ms_handle_connect 0x7fa94c06de40 2024-03-25T18:08:15.163 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.162+0000 7fa9657fa640 1 -- 172.21.15.66:0/3192329527 <== mon.2 v1:172.21.15.66:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7fa9540549d0 con 0x7fa96813f710 2024-03-25T18:08:15.163 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.162+0000 7fa9657fa640 10 client.4421.objecter ms_dispatch 0x7fa968000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:15.163 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.162+0000 7fa9657fa640 3 client.4421.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:15.163 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.162+0000 7fa9657fa640 3 client.4421.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:15.163 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.162+0000 7fa9657fa640 20 client.4421.objecter dump_active .. 0 homeless 2024-03-25T18:08:15.163 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.162+0000 7fa9657fa640 1 -- 172.21.15.66:0/3192329527 <== mon.2 v1:172.21.15.66:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa9540504e0 con 0x7fa96813f710 2024-03-25T18:08:15.163 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.163+0000 7fa9657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:15.163 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.163+0000 7fa9657fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:15.416 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.416+0000 7fa96eeed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:15.417 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.416+0000 7fa96eeed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:15.417 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.416+0000 7fa96eeed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:15.417 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.416+0000 7fa96eeed640 1 -- 172.21.15.66:0/3192329527 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa96814e880 con 0x7fa96813f710 2024-03-25T18:08:15.440 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.439+0000 7fa9657fa640 1 -- 172.21.15.66:0/3192329527 <== mon.2 v1:172.21.15.66:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7fa954052410 con 0x7fa96813f710 2024-03-25T18:08:15.440 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.439+0000 7fa9657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:15.440 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.439+0000 7fa9657fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:15.440 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:08:15.440 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:15.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.442+0000 7fa96eeed640 10 librados: watch_flush enter 2024-03-25T18:08:15.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.442+0000 7fa96eeed640 10 librados: watch_flush exit 2024-03-25T18:08:15.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.442+0000 7fa96eeed640 20 client.4421.objecter shutdown clearing up homeless session... 2024-03-25T18:08:15.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.442+0000 7fa96eeed640 10 client.4421.objecter successfully canceled tick 2024-03-25T18:08:15.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.442+0000 7fa96eeed640 1 -- 172.21.15.66:0/3192329527 >> v1:172.21.15.193:6800/2275676803 conn(0x7fa94c06de40 legacy=0x7fa94c070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:15.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.442+0000 7fa96eeed640 10 monclient: shutdown 2024-03-25T18:08:15.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.442+0000 7fa96eeed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:15.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.442+0000 7fa96eeed640 1 -- 172.21.15.66:0/3192329527 >> v1:172.21.15.66:6790/0 conn(0x7fa96813f710 legacy=0x7fa9681d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:15.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.442+0000 7fa96eeed640 1 -- 172.21.15.66:0/3192329527 shutdown_connections 2024-03-25T18:08:15.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.442+0000 7fa96eeed640 1 -- 172.21.15.66:0/3192329527 >> 172.21.15.66:0/3192329527 conn(0x7fa9680b7ea0 msgr2=0x7fa968074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:15.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.442+0000 7fa96eeed640 1 -- 172.21.15.66:0/3192329527 shutdown_connections 2024-03-25T18:08:15.444 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.442+0000 7fa96eeed640 1 -- 172.21.15.66:0/3192329527 wait complete. 2024-03-25T18:08:15.444 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:15.442+0000 7fa96eeed640 1 librados: shutdown 2024-03-25T18:08:15.455 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:08:22.263 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:22.383 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.383+0000 7f060bb4b640 10 monclient: get_monmap_and_config 2024-03-25T18:08:22.384 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.383+0000 7f060bb4b640 10 monclient: build_initial_monmap 2024-03-25T18:08:22.384 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.383+0000 7f060bb4b640 10 monclient: monmap: 2024-03-25T18:08:22.384 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:08:22.384 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:22.384 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:08:22.384024+0000 2024-03-25T18:08:22.384 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:08:22.384024+0000 2024-03-25T18:08:22.384 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:22.384 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:22.384 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:08:22.384 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:08:22.384 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:08:22.384 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:22.385 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.384+0000 7f060bb4b640 1 Processor -- start 2024-03-25T18:08:22.385 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.384+0000 7f060bb4b640 1 -- start start 2024-03-25T18:08:22.385 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.384+0000 7f060bb4b640 10 monclient: init 2024-03-25T18:08:22.386 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.385+0000 7f060bb4b640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:22.386 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.385+0000 7f060bb4b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:08:22.386 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.385+0000 7f060bb4b640 10 monclient(hunting): picked mon.noname-a con 0x7f06040a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:08:22.386 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.385+0000 7f060bb4b640 10 monclient(hunting): picked mon.noname-b con 0x7f060413f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:08:22.386 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.385+0000 7f060bb4b640 10 monclient(hunting): picked mon.noname-c con 0x7f06041430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:08:22.386 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.385+0000 7f060bb4b640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0604142520 con 0x7f06040a98f0 2024-03-25T18:08:22.387 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.385+0000 7f060bb4b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06040b28a0 con 0x7f06041430c0 2024-03-25T18:08:22.387 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.385+0000 7f060bb4b640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06040b2a30 con 0x7f060413f710 2024-03-25T18:08:22.387 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.385+0000 7f060bb4b640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:22.387 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.385+0000 7f060bb4b640 10 monclient(hunting): authenticate will time out at 7251.444831s 2024-03-25T18:08:22.387 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.387+0000 7f06098e7640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f06040a98f0 0x7f06040a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:42004/0 (socket says 172.21.15.66:42004) 2024-03-25T18:08:22.387 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.387+0000 7f06098e7640 1 -- 172.21.15.66:0/1872164349 learned_addr learned my addr 172.21.15.66:0/1872164349 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:08:22.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.395+0000 7f06088e5640 1 -- 172.21.15.66:0/1872164349 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4090448365 0 0) 0x7f0604142520 con 0x7f06040a98f0 2024-03-25T18:08:22.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.395+0000 7f06088e5640 10 monclient(hunting): _negotiate 2024-03-25T18:08:22.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.395+0000 7f06088e5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:22.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.395+0000 7f06088e5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:22.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.395+0000 7f06088e5640 10 monclient(hunting): my global_id is 4422 2024-03-25T18:08:22.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.395+0000 7f06088e5640 1 -- 172.21.15.66:0/1872164349 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05f0003520 con 0x7f06040a98f0 2024-03-25T18:08:22.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.402+0000 7f06088e5640 1 -- 172.21.15.66:0/1872164349 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2362042048 0 0) 0x7f05f0003520 con 0x7f06040a98f0 2024-03-25T18:08:22.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.402+0000 7f06088e5640 1 -- 172.21.15.66:0/1872164349 >> v1:172.21.15.66:6790/0 conn(0x7f060413f710 legacy=0x7f06041429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:08:22.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.402+0000 7f06088e5640 1 -- 172.21.15.66:0/1872164349 >> v1:172.21.15.193:6789/0 conn(0x7f06041430c0 legacy=0x7f060414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:22.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.402+0000 7f06088e5640 10 monclient: _finish_hunting 0 2024-03-25T18:08:22.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.402+0000 7f06088e5640 1 monclient: found mon.noname-a 2024-03-25T18:08:22.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.402+0000 7f06088e5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:22.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.402+0000 7f06088e5640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.66:6789/0 2024-03-25T18:08:22.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.402+0000 7f06088e5640 1 -- 172.21.15.66:0/1872164349 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06040b44c0 con 0x7f06040a98f0 2024-03-25T18:08:22.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.402+0000 7f06088e5640 10 monclient: _finish_auth 0 2024-03-25T18:08:22.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.402+0000 7f06088e5640 10 monclient: _check_auth_tickets 2024-03-25T18:08:22.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.402+0000 7f06088e5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:22.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.402+0000 7f06088e5640 1 -- 172.21.15.66:0/1872164349 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f0600001690 con 0x7f06040a98f0 2024-03-25T18:08:22.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.402+0000 7f06088e5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:22.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.402+0000 7f06088e5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:08:22.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.402+0000 7f06088e5640 10 monclient: dump: 2024-03-25T18:08:22.404 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:22.404 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:22.404 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:22.404 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:22.404 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:22.404 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:22.404 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:22.404 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:22.404 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:22.404 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:22.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.402+0000 7f060bb4b640 5 monclient: authenticate success, global_id 4422 2024-03-25T18:08:22.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.402+0000 7f060bb4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:22.409 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.409+0000 7f06088e5640 1 -- 172.21.15.66:0/1872164349 <== mon.0 v1:172.21.15.66:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0600001860 con 0x7f06040a98f0 2024-03-25T18:08:22.409 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.409+0000 7f06088e5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:22.409 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.409+0000 7f06088e5640 1 -- 172.21.15.66:0/1872164349 <== mon.0 v1:172.21.15.66:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f0600001cb0 con 0x7f06040a98f0 2024-03-25T18:08:22.409 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.409+0000 7f060bb4b640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:22.409 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.409+0000 7f060bb4b640 4 set_mon_vals no callback set 2024-03-25T18:08:22.409 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.409+0000 7f06088e5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:22.410 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.409+0000 7f06088e5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:22.410 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.409+0000 7f06088e5640 10 monclient: dump: 2024-03-25T18:08:22.410 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:22.410 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:22.410 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:22.410 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:22.410 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:22.410 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:22.410 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:22.410 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:22.410 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:22.410 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:22.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.410+0000 7f060bb4b640 10 monclient: shutdown 2024-03-25T18:08:22.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.410+0000 7f060bb4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:22.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.410+0000 7f060bb4b640 1 -- 172.21.15.66:0/1872164349 >> v1:172.21.15.66:6789/0 conn(0x7f06040a98f0 legacy=0x7f06040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:22.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.410+0000 7f060bb4b640 1 -- 172.21.15.66:0/1872164349 shutdown_connections 2024-03-25T18:08:22.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.410+0000 7f060bb4b640 1 -- 172.21.15.66:0/1872164349 >> 172.21.15.66:0/1872164349 conn(0x7f06040b7ea0 msgr2=0x7f06040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:22.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.410+0000 7f060bb4b640 1 -- 172.21.15.66:0/1872164349 shutdown_connections 2024-03-25T18:08:22.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.410+0000 7f060bb4b640 1 -- 172.21.15.66:0/1872164349 wait complete. 2024-03-25T18:08:22.412 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.411+0000 7f060bb4b640 10 monclient: build_initial_monmap 2024-03-25T18:08:22.412 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.411+0000 7f060bb4b640 10 monclient: monmap: 2024-03-25T18:08:22.412 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:08:22.412 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:22.413 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:08:22.413 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:08:22.413 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:22.413 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:08:22.413 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:08:22.413 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:08:22.413 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:08:22.413 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:22.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.412+0000 7f060bb4b640 1 librados: starting msgr at 2024-03-25T18:08:22.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.412+0000 7f060bb4b640 1 librados: starting objecter 2024-03-25T18:08:22.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.412+0000 7f060bb4b640 1 Processor -- start 2024-03-25T18:08:22.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.412+0000 7f060bb4b640 1 -- start start 2024-03-25T18:08:22.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.412+0000 7f060bb4b640 1 librados: setting wanted keys 2024-03-25T18:08:22.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.412+0000 7f060bb4b640 1 librados: calling monclient init 2024-03-25T18:08:22.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.412+0000 7f060bb4b640 10 monclient: init 2024-03-25T18:08:22.414 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.413+0000 7f060bb4b640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:22.415 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.413+0000 7f060bb4b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:08:22.415 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.413+0000 7f060bb4b640 10 monclient(hunting): picked mon.noname-b con 0x7f06040a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:08:22.415 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.413+0000 7f060bb4b640 10 monclient(hunting): picked mon.noname-c con 0x7f060413f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:08:22.415 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.413+0000 7f060bb4b640 10 monclient(hunting): picked mon.noname-a con 0x7f06041430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:08:22.415 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.413+0000 7f060bb4b640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0604136f30 con 0x7f06041430c0 2024-03-25T18:08:22.415 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.413+0000 7f060bb4b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0604135570 con 0x7f06040a98f0 2024-03-25T18:08:22.415 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.413+0000 7f060bb4b640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0604135070 con 0x7f060413f710 2024-03-25T18:08:22.415 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.413+0000 7f060bb4b640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:22.415 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.413+0000 7f060bb4b640 10 monclient(hunting): authenticate will time out at 7251.472624s 2024-03-25T18:08:22.417 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.417+0000 7f06098e7640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f06040a98f0 0x7f06041d4790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:43326/0 (socket says 172.21.15.66:43326) 2024-03-25T18:08:22.417 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.417+0000 7f06098e7640 1 -- 172.21.15.66:0/1276824415 learned_addr learned my addr 172.21.15.66:0/1276824415 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:08:22.419 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.419+0000 7f05fa7fc640 10 client.?.objecter ms_handle_connect 0x7f06040a98f0 2024-03-25T18:08:22.420 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.419+0000 7f05fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:22.423 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.423+0000 7f05fa7fc640 10 client.?.objecter ms_handle_connect 0x7f060413f710 2024-03-25T18:08:22.423 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.423+0000 7f05fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:22.424 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.424+0000 7f05fa7fc640 10 client.?.objecter ms_handle_connect 0x7f06041430c0 2024-03-25T18:08:22.424 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.424+0000 7f05fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:22.426 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.425+0000 7f05fa7fc640 1 -- 172.21.15.66:0/1276824415 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 335563964 0 0) 0x7f0604135570 con 0x7f06040a98f0 2024-03-25T18:08:22.426 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.425+0000 7f05fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:22.426 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.425+0000 7f05fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:22.426 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.425+0000 7f05fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:22.426 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.425+0000 7f05fa7fc640 10 monclient(hunting): my global_id is 4423 2024-03-25T18:08:22.426 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.425+0000 7f05fa7fc640 1 -- 172.21.15.66:0/1276824415 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05e8004eb0 con 0x7f06040a98f0 2024-03-25T18:08:22.428 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.428+0000 7f05fa7fc640 1 -- 172.21.15.66:0/1276824415 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4284484072 0 0) 0x7f0604135070 con 0x7f060413f710 2024-03-25T18:08:22.428 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.428+0000 7f05fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:22.428 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.428+0000 7f05fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:22.429 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.428+0000 7f05fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:22.429 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.428+0000 7f05fa7fc640 10 monclient(hunting): my global_id is 4424 2024-03-25T18:08:22.429 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.428+0000 7f05fa7fc640 1 -- 172.21.15.66:0/1276824415 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0604135570 con 0x7f060413f710 2024-03-25T18:08:22.433 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.433+0000 7f05fa7fc640 1 -- 172.21.15.66:0/1276824415 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 332454078 0 0) 0x7f0604136f30 con 0x7f06041430c0 2024-03-25T18:08:22.433 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.433+0000 7f05fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:08:22.433 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.433+0000 7f05fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:22.433 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.433+0000 7f05fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:22.433 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.433+0000 7f05fa7fc640 10 monclient(hunting): my global_id is 4425 2024-03-25T18:08:22.433 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.433+0000 7f05fa7fc640 1 -- 172.21.15.66:0/1276824415 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0604135070 con 0x7f06041430c0 2024-03-25T18:08:22.435 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.434+0000 7f05fa7fc640 1 -- 172.21.15.66:0/1276824415 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3487297566 0 0) 0x7f0604135570 con 0x7f060413f710 2024-03-25T18:08:22.435 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.434+0000 7f05fa7fc640 1 -- 172.21.15.66:0/1276824415 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0604136f30 con 0x7f060413f710 2024-03-25T18:08:22.435 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.434+0000 7f05fa7fc640 1 -- 172.21.15.66:0/1276824415 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f05f4001810 con 0x7f060413f710 2024-03-25T18:08:22.435 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.434+0000 7f05fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:22.435 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.434+0000 7f05fa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:22.435 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.434+0000 7f05fa7fc640 10 monclient(hunting): dump: 2024-03-25T18:08:22.435 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:22.435 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:22.435 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:22.436 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:22.436 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:22.436 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:22.436 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:22.436 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:22.436 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:22.436 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:22.436 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.434+0000 7f05fa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:22.442 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.442+0000 7f05fa7fc640 1 -- 172.21.15.66:0/1276824415 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1690818398 0 0) 0x7f05e8004eb0 con 0x7f06040a98f0 2024-03-25T18:08:22.442 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.442+0000 7f05fa7fc640 1 -- 172.21.15.66:0/1276824415 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0604135570 con 0x7f06040a98f0 2024-03-25T18:08:22.442 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.442+0000 7f05fa7fc640 1 -- 172.21.15.66:0/1276824415 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f0600001b70 con 0x7f06040a98f0 2024-03-25T18:08:22.442 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.442+0000 7f05fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:22.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.442+0000 7f05fa7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:22.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.442+0000 7f05fa7fc640 10 monclient(hunting): dump: 2024-03-25T18:08:22.443 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:22.443 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:22.443 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:22.443 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:22.443 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:22.443 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:22.443 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:22.443 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:22.443 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:22.443 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:22.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.446+0000 7f05fa7fc640 1 -- 172.21.15.66:0/1276824415 <== mon.2 v1:172.21.15.66:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3436476689 0 0) 0x7f0604136f30 con 0x7f060413f710 2024-03-25T18:08:22.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.446+0000 7f05fa7fc640 1 -- 172.21.15.66:0/1276824415 >> v1:172.21.15.193:6789/0 conn(0x7f06040a98f0 legacy=0x7f06041d4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:22.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.446+0000 7f05fa7fc640 1 -- 172.21.15.66:0/1276824415 >> v1:172.21.15.66:6789/0 conn(0x7f06041430c0 legacy=0x7f06041db4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:22.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.446+0000 7f05fa7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:08:22.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.446+0000 7f05fa7fc640 1 monclient: found mon.c 2024-03-25T18:08:22.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.446+0000 7f05fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:22.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.446+0000 7f05fa7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:22.448 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.446+0000 7f05fa7fc640 1 -- 172.21.15.66:0/1276824415 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06041d41e0 con 0x7f060413f710 2024-03-25T18:08:22.448 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.446+0000 7f05fa7fc640 10 monclient: _finish_auth 0 2024-03-25T18:08:22.448 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.446+0000 7f05fa7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:08:22.448 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.446+0000 7f05fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:22.448 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.446+0000 7f060bb4b640 5 monclient: authenticate success, global_id 4424 2024-03-25T18:08:22.448 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.446+0000 7f060bb4b640 10 monclient: _renew_subs 2024-03-25T18:08:22.448 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.446+0000 7f060bb4b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:22.448 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.446+0000 7f060bb4b640 1 -- 172.21.15.66:0/1276824415 --> v1:172.21.15.66:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0604135cf0 con 0x7f060413f710 2024-03-25T18:08:22.448 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.446+0000 7f060bb4b640 10 client.4424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:22.448 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.446+0000 7f060bb4b640 10 monclient: _renew_subs 2024-03-25T18:08:22.448 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.446+0000 7f060bb4b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:22.448 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.446+0000 7f060bb4b640 1 -- 172.21.15.66:0/1276824415 --> v1:172.21.15.66:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f060413ca50 con 0x7f060413f710 2024-03-25T18:08:22.448 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.446+0000 7f060bb4b640 1 librados: init done 2024-03-25T18:08:22.449 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.448+0000 7f060bb4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:22.449 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.448+0000 7f060bb4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:22.449 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.448+0000 7f060bb4b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:22.449 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.448+0000 7f060bb4b640 1 -- 172.21.15.66:0/1276824415 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06041d43d0 con 0x7f060413f710 2024-03-25T18:08:22.766 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.766+0000 7f05fa7fc640 1 -- 172.21.15.66:0/1276824415 <== mon.2 v1:172.21.15.66:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05f4001d00 con 0x7f060413f710 2024-03-25T18:08:22.767 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.766+0000 7f05fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:22.767 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.766+0000 7f05fa7fc640 1 -- 172.21.15.66:0/1276824415 <== mon.2 v1:172.21.15.66:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f05f4003790 con 0x7f060413f710 2024-03-25T18:08:22.767 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.766+0000 7f05fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:22.767 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.766+0000 7f05fa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:22.767 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.766+0000 7f05fa7fc640 10 monclient: dump: 2024-03-25T18:08:22.767 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:22.767 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:22.767 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:22.767 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:22.767 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:22.767 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:22.767 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:22.767 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:22.767 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:22.767 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:22.768 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.766+0000 7f05fbfff640 4 set_mon_vals no callback set 2024-03-25T18:08:22.768 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.766+0000 7f05fa7fc640 1 -- 172.21.15.66:0/1276824415 <== mon.2 v1:172.21.15.66:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f05f40197a0 con 0x7f060413f710 2024-03-25T18:08:22.768 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.767+0000 7f06098e7640 10 monclient: get_auth_request con 0x7f05e806df10 auth_method 0 2024-03-25T18:08:22.768 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.767+0000 7f05fa7fc640 10 client.4424.objecter ms_handle_connect 0x7f05e806df10 2024-03-25T18:08:22.775 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.774+0000 7f05fa7fc640 1 -- 172.21.15.66:0/1276824415 <== mon.2 v1:172.21.15.66:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f05f4053470 con 0x7f060413f710 2024-03-25T18:08:22.775 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.774+0000 7f05fa7fc640 10 client.4424.objecter ms_dispatch 0x7f0604000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:22.775 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.774+0000 7f05fa7fc640 3 client.4424.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:22.775 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.774+0000 7f05fa7fc640 3 client.4424.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:22.775 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.774+0000 7f05fa7fc640 20 client.4424.objecter dump_active .. 0 homeless 2024-03-25T18:08:22.775 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.774+0000 7f05fa7fc640 1 -- 172.21.15.66:0/1276824415 <== mon.2 v1:172.21.15.66:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f05f404ef80 con 0x7f060413f710 2024-03-25T18:08:22.775 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.774+0000 7f05fa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:22.775 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:22.774+0000 7f05fa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:23.015 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:23.015+0000 7f060bb4b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:23.015 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:23.015+0000 7f060bb4b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:23.016 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:23.015+0000 7f060bb4b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:23.016 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:23.015+0000 7f060bb4b640 1 -- 172.21.15.66:0/1276824415 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06040b3ef0 con 0x7f060413f710 2024-03-25T18:08:23.063 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:23.063+0000 7f05fa7fc640 1 -- 172.21.15.66:0/1276824415 <== mon.2 v1:172.21.15.66:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f05f4050eb0 con 0x7f060413f710 2024-03-25T18:08:23.063 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:23.063+0000 7f05fa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:23.063 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:23.063+0000 7f05fa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:23.064 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:08:23.064 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:23.066 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:23.064+0000 7f060bb4b640 10 librados: watch_flush enter 2024-03-25T18:08:23.066 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:23.065+0000 7f060bb4b640 10 librados: watch_flush exit 2024-03-25T18:08:23.066 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:23.065+0000 7f060bb4b640 20 client.4424.objecter shutdown clearing up homeless session... 2024-03-25T18:08:23.066 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:23.065+0000 7f060bb4b640 10 client.4424.objecter successfully canceled tick 2024-03-25T18:08:23.066 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:23.065+0000 7f060bb4b640 1 -- 172.21.15.66:0/1276824415 >> v1:172.21.15.193:6800/2275676803 conn(0x7f05e806df10 legacy=0x7f05e80703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:23.066 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:23.065+0000 7f060bb4b640 10 monclient: shutdown 2024-03-25T18:08:23.066 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:23.065+0000 7f060bb4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:23.066 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:23.065+0000 7f060bb4b640 1 -- 172.21.15.66:0/1276824415 >> v1:172.21.15.66:6790/0 conn(0x7f060413f710 legacy=0x7f06041dade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:23.066 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:23.065+0000 7f060bb4b640 1 -- 172.21.15.66:0/1276824415 shutdown_connections 2024-03-25T18:08:23.066 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:23.065+0000 7f060bb4b640 1 -- 172.21.15.66:0/1276824415 >> 172.21.15.66:0/1276824415 conn(0x7f06040b7ea0 msgr2=0x7f0604074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:23.067 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:23.065+0000 7f060bb4b640 1 -- 172.21.15.66:0/1276824415 shutdown_connections 2024-03-25T18:08:23.067 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:23.065+0000 7f060bb4b640 1 -- 172.21.15.66:0/1276824415 wait complete. 2024-03-25T18:08:23.067 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:23.065+0000 7f060bb4b640 1 librados: shutdown 2024-03-25T18:08:23.077 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:08:24.403 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:04:55.383 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:24.412 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:04:55.393 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:24.454 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:04:55.434 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:24.465 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:04:55.423 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:24.470 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:04:55.429 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:24.475 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:04:55.456 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:24.491 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:04:55.471 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:24.501 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:04:55.460 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:24.523 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:04:55.504 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:29.883 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:30.000 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:29.998+0000 7f5b2aef8640 10 monclient: get_monmap_and_config 2024-03-25T18:08:30.000 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:29.998+0000 7f5b2aef8640 10 monclient: build_initial_monmap 2024-03-25T18:08:30.000 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:29.998+0000 7f5b2aef8640 10 monclient: monmap: 2024-03-25T18:08:30.000 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:08:30.001 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:30.001 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:08:30.000346+0000 2024-03-25T18:08:30.001 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:08:30.000346+0000 2024-03-25T18:08:30.001 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:30.001 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:30.001 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:08:30.001 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:08:30.001 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:08:30.001 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:30.003 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:29.999+0000 7f5b2aef8640 1 Processor -- start 2024-03-25T18:08:30.003 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:29.999+0000 7f5b2aef8640 1 -- start start 2024-03-25T18:08:30.003 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:29.999+0000 7f5b2aef8640 10 monclient: init 2024-03-25T18:08:30.003 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.000+0000 7f5b2aef8640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:30.003 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.000+0000 7f5b2aef8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:08:30.003 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.000+0000 7f5b2aef8640 10 monclient(hunting): picked mon.noname-b con 0x7f5b240a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:08:30.003 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.000+0000 7f5b2aef8640 10 monclient(hunting): picked mon.noname-a con 0x7f5b2413f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:08:30.003 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.000+0000 7f5b2aef8640 10 monclient(hunting): picked mon.noname-c con 0x7f5b241430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:08:30.003 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.000+0000 7f5b2aef8640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b24142520 con 0x7f5b2413f710 2024-03-25T18:08:30.003 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.000+0000 7f5b2aef8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b240b28a0 con 0x7f5b241430c0 2024-03-25T18:08:30.003 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.000+0000 7f5b2aef8640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b240b2a30 con 0x7f5b240a98f0 2024-03-25T18:08:30.003 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.000+0000 7f5b2aef8640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:30.004 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.000+0000 7f5b2aef8640 10 monclient(hunting): authenticate will time out at 7259.060649s 2024-03-25T18:08:30.016 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.015+0000 7f5b29495640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f5b241430c0 0x7f5b2414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:44226/0 (socket says 172.21.15.66:44226) 2024-03-25T18:08:30.016 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.015+0000 7f5b29495640 1 -- 172.21.15.66:0/3029192685 learned_addr learned my addr 172.21.15.66:0/3029192685 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:08:30.032 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.030+0000 7f5b237fe640 1 -- 172.21.15.66:0/3029192685 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052416441 0 0) 0x7f5b240b28a0 con 0x7f5b241430c0 2024-03-25T18:08:30.032 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.030+0000 7f5b237fe640 10 monclient(hunting): _negotiate 2024-03-25T18:08:30.032 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.030+0000 7f5b237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:30.032 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.030+0000 7f5b237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:30.032 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.030+0000 7f5b237fe640 10 monclient(hunting): my global_id is 4426 2024-03-25T18:08:30.032 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.030+0000 7f5b237fe640 1 -- 172.21.15.66:0/3029192685 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b14003520 con 0x7f5b241430c0 2024-03-25T18:08:30.035 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.033+0000 7f5b237fe640 1 -- 172.21.15.66:0/3029192685 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1101636083 0 0) 0x7f5b240b2a30 con 0x7f5b240a98f0 2024-03-25T18:08:30.035 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.033+0000 7f5b237fe640 10 monclient(hunting): _negotiate 2024-03-25T18:08:30.035 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.033+0000 7f5b237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:30.035 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.033+0000 7f5b237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:30.035 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.033+0000 7f5b237fe640 10 monclient(hunting): my global_id is 4427 2024-03-25T18:08:30.035 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.033+0000 7f5b237fe640 1 -- 172.21.15.66:0/3029192685 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b240b28a0 con 0x7f5b240a98f0 2024-03-25T18:08:30.040 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.039+0000 7f5b237fe640 1 -- 172.21.15.66:0/3029192685 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 213961764 0 0) 0x7f5b240b28a0 con 0x7f5b240a98f0 2024-03-25T18:08:30.040 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.039+0000 7f5b237fe640 1 -- 172.21.15.66:0/3029192685 >> v1:172.21.15.193:6789/0 conn(0x7f5b241430c0 legacy=0x7f5b2414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:30.040 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.039+0000 7f5b237fe640 1 -- 172.21.15.66:0/3029192685 >> v1:172.21.15.66:6789/0 conn(0x7f5b2413f710 legacy=0x7f5b241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:30.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.039+0000 7f5b237fe640 10 monclient: _finish_hunting 0 2024-03-25T18:08:30.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.039+0000 7f5b237fe640 1 monclient: found mon.noname-b 2024-03-25T18:08:30.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.039+0000 7f5b237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:30.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.039+0000 7f5b237fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.66:6790/0 2024-03-25T18:08:30.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.039+0000 7f5b237fe640 1 -- 172.21.15.66:0/3029192685 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b240b44c0 con 0x7f5b240a98f0 2024-03-25T18:08:30.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.039+0000 7f5b237fe640 10 monclient: _finish_auth 0 2024-03-25T18:08:30.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.039+0000 7f5b237fe640 10 monclient: _check_auth_tickets 2024-03-25T18:08:30.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.039+0000 7f5b237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:30.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.039+0000 7f5b237fe640 1 -- 172.21.15.66:0/3029192685 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f5b18001670 con 0x7f5b240a98f0 2024-03-25T18:08:30.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.039+0000 7f5b2aef8640 5 monclient: authenticate success, global_id 4427 2024-03-25T18:08:30.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.039+0000 7f5b2aef8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:30.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.039+0000 7f5b237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:30.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.039+0000 7f5b237fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:08:30.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.039+0000 7f5b237fe640 10 monclient: dump: 2024-03-25T18:08:30.041 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:30.042 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:30.042 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:30.042 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:30.042 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:30.042 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:30.042 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:30.042 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:30.042 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:30.042 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:30.042 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.039+0000 7f5b2aef8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:30.046 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.045+0000 7f5b237fe640 1 -- 172.21.15.66:0/3029192685 <== mon.2 v1:172.21.15.66:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b180017d0 con 0x7f5b240a98f0 2024-03-25T18:08:30.046 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.045+0000 7f5b237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:30.046 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.045+0000 7f5b237fe640 1 -- 172.21.15.66:0/3029192685 <== mon.2 v1:172.21.15.66:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f5b18001ba0 con 0x7f5b240a98f0 2024-03-25T18:08:30.047 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.045+0000 7f5b2aef8640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:30.047 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.045+0000 7f5b2aef8640 4 set_mon_vals no callback set 2024-03-25T18:08:30.047 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.045+0000 7f5b237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:30.047 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.045+0000 7f5b237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:30.047 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.045+0000 7f5b237fe640 10 monclient: dump: 2024-03-25T18:08:30.047 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:30.047 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:30.047 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:30.047 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:30.047 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:30.047 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:30.047 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:30.047 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:30.047 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:30.048 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:30.049 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.046+0000 7f5b2aef8640 10 monclient: shutdown 2024-03-25T18:08:30.049 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.046+0000 7f5b2aef8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:30.049 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.046+0000 7f5b2aef8640 1 -- 172.21.15.66:0/3029192685 >> v1:172.21.15.66:6790/0 conn(0x7f5b240a98f0 legacy=0x7f5b240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:30.049 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.046+0000 7f5b2aef8640 1 -- 172.21.15.66:0/3029192685 shutdown_connections 2024-03-25T18:08:30.049 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.046+0000 7f5b2aef8640 1 -- 172.21.15.66:0/3029192685 >> 172.21.15.66:0/3029192685 conn(0x7f5b240b7ea0 msgr2=0x7f5b240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:30.049 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.046+0000 7f5b2aef8640 1 -- 172.21.15.66:0/3029192685 shutdown_connections 2024-03-25T18:08:30.049 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.046+0000 7f5b2aef8640 1 -- 172.21.15.66:0/3029192685 wait complete. 2024-03-25T18:08:30.049 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.046+0000 7f5b2aef8640 10 monclient: build_initial_monmap 2024-03-25T18:08:30.050 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.046+0000 7f5b2aef8640 10 monclient: monmap: 2024-03-25T18:08:30.050 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:08:30.050 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:30.050 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:08:30.050 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:08:30.050 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:30.050 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:08:30.050 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:08:30.050 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:08:30.050 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:08:30.050 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:30.050 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.047+0000 7f5b2aef8640 1 librados: starting msgr at 2024-03-25T18:08:30.050 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.047+0000 7f5b2aef8640 1 librados: starting objecter 2024-03-25T18:08:30.050 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.047+0000 7f5b2aef8640 1 Processor -- start 2024-03-25T18:08:30.050 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.047+0000 7f5b2aef8640 1 -- start start 2024-03-25T18:08:30.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.047+0000 7f5b2aef8640 1 librados: setting wanted keys 2024-03-25T18:08:30.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.047+0000 7f5b2aef8640 1 librados: calling monclient init 2024-03-25T18:08:30.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.047+0000 7f5b2aef8640 10 monclient: init 2024-03-25T18:08:30.052 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.048+0000 7f5b2aef8640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:30.052 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.048+0000 7f5b2aef8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:08:30.052 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.048+0000 7f5b2aef8640 10 monclient(hunting): picked mon.noname-b con 0x7f5b240a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:08:30.052 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.049+0000 7f5b2aef8640 10 monclient(hunting): picked mon.noname-c con 0x7f5b2413f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:08:30.052 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.049+0000 7f5b2aef8640 10 monclient(hunting): picked mon.noname-a con 0x7f5b241430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:08:30.052 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.049+0000 7f5b2aef8640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b241d7870 con 0x7f5b241430c0 2024-03-25T18:08:30.052 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.049+0000 7f5b2aef8640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b240abd10 con 0x7f5b240a98f0 2024-03-25T18:08:30.052 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.049+0000 7f5b2aef8640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b241d7490 con 0x7f5b2413f710 2024-03-25T18:08:30.052 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.049+0000 7f5b2aef8640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:30.052 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.049+0000 7f5b2aef8640 10 monclient(hunting): authenticate will time out at 7259.109225s 2024-03-25T18:08:30.061 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.059+0000 7f5b28c94640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f5b240a98f0 0x7f5b24138710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:44248/0 (socket says 172.21.15.66:44248) 2024-03-25T18:08:30.061 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.059+0000 7f5b28c94640 1 -- 172.21.15.66:0/872271282 learned_addr learned my addr 172.21.15.66:0/872271282 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:08:30.066 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.065+0000 7f5b217fa640 10 client.?.objecter ms_handle_connect 0x7f5b240a98f0 2024-03-25T18:08:30.066 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.065+0000 7f5b217fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:30.071 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.070+0000 7f5b217fa640 10 client.?.objecter ms_handle_connect 0x7f5b241430c0 2024-03-25T18:08:30.071 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.070+0000 7f5b217fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:30.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.071+0000 7f5b217fa640 10 client.?.objecter ms_handle_connect 0x7f5b2413f710 2024-03-25T18:08:30.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.071+0000 7f5b217fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:30.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.076+0000 7f5b217fa640 1 -- 172.21.15.66:0/872271282 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 881773805 0 0) 0x7f5b240abd10 con 0x7f5b240a98f0 2024-03-25T18:08:30.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.076+0000 7f5b217fa640 10 monclient(hunting): _negotiate 2024-03-25T18:08:30.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.076+0000 7f5b217fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:30.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.076+0000 7f5b217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:30.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.076+0000 7f5b217fa640 10 monclient(hunting): my global_id is 4429 2024-03-25T18:08:30.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.076+0000 7f5b217fa640 1 -- 172.21.15.66:0/872271282 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b08004eb0 con 0x7f5b240a98f0 2024-03-25T18:08:30.084 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.082+0000 7f5b217fa640 1 -- 172.21.15.66:0/872271282 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4221105995 0 0) 0x7f5b241d7490 con 0x7f5b2413f710 2024-03-25T18:08:30.084 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.082+0000 7f5b217fa640 10 monclient(hunting): _negotiate 2024-03-25T18:08:30.084 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.082+0000 7f5b217fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:30.084 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.082+0000 7f5b217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:30.084 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.082+0000 7f5b217fa640 10 monclient(hunting): my global_id is 4430 2024-03-25T18:08:30.084 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.082+0000 7f5b217fa640 1 -- 172.21.15.66:0/872271282 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b240abd10 con 0x7f5b2413f710 2024-03-25T18:08:30.088 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.087+0000 7f5b217fa640 1 -- 172.21.15.66:0/872271282 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1851167182 0 0) 0x7f5b241d7870 con 0x7f5b241430c0 2024-03-25T18:08:30.088 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.087+0000 7f5b217fa640 10 monclient(hunting): _negotiate 2024-03-25T18:08:30.088 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.087+0000 7f5b217fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:30.088 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.087+0000 7f5b217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:30.088 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.087+0000 7f5b217fa640 10 monclient(hunting): my global_id is 4431 2024-03-25T18:08:30.088 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.087+0000 7f5b217fa640 1 -- 172.21.15.66:0/872271282 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b241d7490 con 0x7f5b241430c0 2024-03-25T18:08:30.093 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.091+0000 7f5b217fa640 1 -- 172.21.15.66:0/872271282 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2845231435 0 0) 0x7f5b240abd10 con 0x7f5b2413f710 2024-03-25T18:08:30.093 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.091+0000 7f5b217fa640 1 -- 172.21.15.66:0/872271282 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b241d7870 con 0x7f5b2413f710 2024-03-25T18:08:30.093 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.091+0000 7f5b217fa640 1 -- 172.21.15.66:0/872271282 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f5b100018a0 con 0x7f5b2413f710 2024-03-25T18:08:30.093 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.091+0000 7f5b217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:30.093 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.091+0000 7f5b217fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:30.093 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.091+0000 7f5b217fa640 10 monclient(hunting): dump: 2024-03-25T18:08:30.093 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:30.093 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:30.093 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:30.094 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:30.094 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:30.094 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:30.094 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:30.094 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:30.094 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:30.094 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:30.094 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.092+0000 7f5b217fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:30.099 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.098+0000 7f5b217fa640 1 -- 172.21.15.66:0/872271282 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2158207402 0 0) 0x7f5b08004eb0 con 0x7f5b240a98f0 2024-03-25T18:08:30.099 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.098+0000 7f5b217fa640 1 -- 172.21.15.66:0/872271282 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b240abd10 con 0x7f5b240a98f0 2024-03-25T18:08:30.099 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.098+0000 7f5b217fa640 1 -- 172.21.15.66:0/872271282 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f5b18001be0 con 0x7f5b240a98f0 2024-03-25T18:08:30.099 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.098+0000 7f5b217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:30.100 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.098+0000 7f5b217fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:30.100 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.098+0000 7f5b217fa640 10 monclient(hunting): dump: 2024-03-25T18:08:30.100 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:30.100 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:30.100 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:30.100 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:30.100 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:30.100 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:30.100 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:30.100 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:30.100 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:30.100 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:30.109 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.107+0000 7f5b217fa640 1 -- 172.21.15.66:0/872271282 <== mon.2 v1:172.21.15.66:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2948064592 0 0) 0x7f5b241d7870 con 0x7f5b2413f710 2024-03-25T18:08:30.109 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.107+0000 7f5b217fa640 1 -- 172.21.15.66:0/872271282 >> v1:172.21.15.193:6789/0 conn(0x7f5b240a98f0 legacy=0x7f5b24138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:30.109 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.107+0000 7f5b217fa640 1 -- 172.21.15.66:0/872271282 >> v1:172.21.15.66:6789/0 conn(0x7f5b241430c0 legacy=0x7f5b24134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:30.109 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.107+0000 7f5b217fa640 10 monclient: _finish_hunting 0 2024-03-25T18:08:30.109 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.107+0000 7f5b217fa640 1 monclient: found mon.c 2024-03-25T18:08:30.109 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.107+0000 7f5b217fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:30.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.107+0000 7f5b217fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:30.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.107+0000 7f5b217fa640 1 -- 172.21.15.66:0/872271282 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b24137a20 con 0x7f5b2413f710 2024-03-25T18:08:30.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.107+0000 7f5b217fa640 10 monclient: _finish_auth 0 2024-03-25T18:08:30.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.107+0000 7f5b217fa640 10 monclient: _check_auth_tickets 2024-03-25T18:08:30.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.107+0000 7f5b217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:30.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.107+0000 7f5b2aef8640 5 monclient: authenticate success, global_id 4430 2024-03-25T18:08:30.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.107+0000 7f5b2aef8640 10 monclient: _renew_subs 2024-03-25T18:08:30.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.107+0000 7f5b2aef8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:30.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.107+0000 7f5b2aef8640 1 -- 172.21.15.66:0/872271282 --> v1:172.21.15.66:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b24137c90 con 0x7f5b2413f710 2024-03-25T18:08:30.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.107+0000 7f5b2aef8640 10 client.4430.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:30.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.107+0000 7f5b2aef8640 10 monclient: _renew_subs 2024-03-25T18:08:30.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.107+0000 7f5b2aef8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:30.111 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.107+0000 7f5b2aef8640 1 -- 172.21.15.66:0/872271282 --> v1:172.21.15.66:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b24134900 con 0x7f5b2413f710 2024-03-25T18:08:30.111 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.107+0000 7f5b2aef8640 1 librados: init done 2024-03-25T18:08:30.111 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.108+0000 7f5b2aef8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:30.111 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.108+0000 7f5b2aef8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:30.111 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.108+0000 7f5b2aef8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:30.111 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.108+0000 7f5b2aef8640 1 -- 172.21.15.66:0/872271282 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b241d7620 con 0x7f5b2413f710 2024-03-25T18:08:30.442 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.440+0000 7f5b217fa640 1 -- 172.21.15.66:0/872271282 <== mon.2 v1:172.21.15.66:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b10001d50 con 0x7f5b2413f710 2024-03-25T18:08:30.442 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.440+0000 7f5b217fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:30.442 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.440+0000 7f5b217fa640 1 -- 172.21.15.66:0/872271282 <== mon.2 v1:172.21.15.66:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f5b10003790 con 0x7f5b2413f710 2024-03-25T18:08:30.442 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.440+0000 7f5b217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:30.442 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.440+0000 7f5b217fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:30.442 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.440+0000 7f5b217fa640 10 monclient: dump: 2024-03-25T18:08:30.442 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:30.443 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:30.443 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:30.443 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:30.443 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:30.443 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:30.443 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:30.443 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:30.443 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:30.443 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:30.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.440+0000 7f5b22ffd640 4 set_mon_vals no callback set 2024-03-25T18:08:30.444 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.441+0000 7f5b217fa640 1 -- 172.21.15.66:0/872271282 <== mon.2 v1:172.21.15.66:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f5b100039b0 con 0x7f5b2413f710 2024-03-25T18:08:30.444 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.442+0000 7f5b28c94640 10 monclient: get_auth_request con 0x7f5b0806df10 auth_method 0 2024-03-25T18:08:30.444 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.442+0000 7f5b217fa640 10 client.4430.objecter ms_handle_connect 0x7f5b0806df10 2024-03-25T18:08:30.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.448+0000 7f5b217fa640 1 -- 172.21.15.66:0/872271282 <== mon.2 v1:172.21.15.66:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f5b10053500 con 0x7f5b2413f710 2024-03-25T18:08:30.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.448+0000 7f5b217fa640 10 client.4430.objecter ms_dispatch 0x7f5b24000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:30.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.448+0000 7f5b217fa640 3 client.4430.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:30.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.448+0000 7f5b217fa640 3 client.4430.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:30.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.448+0000 7f5b217fa640 20 client.4430.objecter dump_active .. 0 homeless 2024-03-25T18:08:30.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.448+0000 7f5b217fa640 1 -- 172.21.15.66:0/872271282 <== mon.2 v1:172.21.15.66:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5b1004ee40 con 0x7f5b2413f710 2024-03-25T18:08:30.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.448+0000 7f5b217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:30.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.448+0000 7f5b217fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:30.712 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.711+0000 7f5b2aef8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:30.712 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.711+0000 7f5b2aef8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:30.712 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.711+0000 7f5b2aef8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:30.712 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.711+0000 7f5b2aef8640 1 -- 172.21.15.66:0/872271282 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b241d7a20 con 0x7f5b2413f710 2024-03-25T18:08:30.844 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.842+0000 7f5b217fa640 1 -- 172.21.15.66:0/872271282 <== mon.2 v1:172.21.15.66:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f5b241d7a20 con 0x7f5b2413f710 2024-03-25T18:08:30.844 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.842+0000 7f5b217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:30.844 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.842+0000 7f5b217fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:30.844 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:08:30.844 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:30.847 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.844+0000 7f5b2aef8640 10 librados: watch_flush enter 2024-03-25T18:08:30.847 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.844+0000 7f5b2aef8640 10 librados: watch_flush exit 2024-03-25T18:08:30.847 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.844+0000 7f5b2aef8640 20 client.4430.objecter shutdown clearing up homeless session... 2024-03-25T18:08:30.847 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.844+0000 7f5b2aef8640 10 client.4430.objecter successfully canceled tick 2024-03-25T18:08:30.847 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.845+0000 7f5b2aef8640 1 -- 172.21.15.66:0/872271282 >> v1:172.21.15.193:6800/2275676803 conn(0x7f5b0806df10 legacy=0x7f5b080703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:30.847 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.845+0000 7f5b2aef8640 10 monclient: shutdown 2024-03-25T18:08:30.847 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.845+0000 7f5b2aef8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:30.847 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.845+0000 7f5b2aef8640 1 -- 172.21.15.66:0/872271282 >> v1:172.21.15.66:6790/0 conn(0x7f5b2413f710 legacy=0x7f5b241d7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:30.847 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.845+0000 7f5b2aef8640 1 -- 172.21.15.66:0/872271282 shutdown_connections 2024-03-25T18:08:30.847 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.845+0000 7f5b2aef8640 1 -- 172.21.15.66:0/872271282 >> 172.21.15.66:0/872271282 conn(0x7f5b240b7ea0 msgr2=0x7f5b24074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:30.847 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.845+0000 7f5b2aef8640 1 -- 172.21.15.66:0/872271282 shutdown_connections 2024-03-25T18:08:30.847 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.845+0000 7f5b2aef8640 1 -- 172.21.15.66:0/872271282 wait complete. 2024-03-25T18:08:30.847 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:30.845+0000 7f5b2aef8640 1 librados: shutdown 2024-03-25T18:08:30.857 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:08:37.664 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:37.781 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.779+0000 7f80db715640 10 monclient: get_monmap_and_config 2024-03-25T18:08:37.781 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.779+0000 7f80db715640 10 monclient: build_initial_monmap 2024-03-25T18:08:37.781 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.779+0000 7f80db715640 10 monclient: monmap: 2024-03-25T18:08:37.781 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:08:37.781 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:37.781 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:08:37.781348+0000 2024-03-25T18:08:37.781 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:08:37.781348+0000 2024-03-25T18:08:37.782 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:37.782 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:37.782 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:08:37.782 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:08:37.782 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:08:37.782 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:37.783 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.780+0000 7f80db715640 1 Processor -- start 2024-03-25T18:08:37.783 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.780+0000 7f80db715640 1 -- start start 2024-03-25T18:08:37.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.780+0000 7f80db715640 10 monclient: init 2024-03-25T18:08:37.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.781+0000 7f80db715640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:37.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.781+0000 7f80db715640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:08:37.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.781+0000 7f80db715640 10 monclient(hunting): picked mon.noname-a con 0x7f80d40b4d60 addr v1:172.21.15.66:6789/0 2024-03-25T18:08:37.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.781+0000 7f80db715640 10 monclient(hunting): picked mon.noname-c con 0x7f80d40b1730 addr v1:172.21.15.193:6789/0 2024-03-25T18:08:37.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.781+0000 7f80db715640 10 monclient(hunting): picked mon.noname-b con 0x7f80d414be90 addr v1:172.21.15.66:6790/0 2024-03-25T18:08:37.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.781+0000 7f80db715640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d40af940 con 0x7f80d40b4d60 2024-03-25T18:08:37.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.781+0000 7f80db715640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d40b0a70 con 0x7f80d40b1730 2024-03-25T18:08:37.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.781+0000 7f80db715640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d40b0d50 con 0x7f80d414be90 2024-03-25T18:08:37.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.781+0000 7f80db715640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:37.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.781+0000 7f80db715640 10 monclient(hunting): authenticate will time out at 7266.841947s 2024-03-25T18:08:37.793 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.792+0000 7f80d8cb0640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f80d40b1730 0x7f80d40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:43024/0 (socket says 172.21.15.66:43024) 2024-03-25T18:08:37.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.792+0000 7f80d8cb0640 1 -- 172.21.15.66:0/2122920969 learned_addr learned my addr 172.21.15.66:0/2122920969 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:08:37.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.810+0000 7f80cbfff640 1 -- 172.21.15.66:0/2122920969 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2305260206 0 0) 0x7f80d40b0a70 con 0x7f80d40b1730 2024-03-25T18:08:37.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.810+0000 7f80cbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:08:37.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.810+0000 7f80cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:37.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.810+0000 7f80cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:37.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.810+0000 7f80cbfff640 10 monclient(hunting): my global_id is 4432 2024-03-25T18:08:37.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.810+0000 7f80cbfff640 1 -- 172.21.15.66:0/2122920969 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80c0003520 con 0x7f80d40b1730 2024-03-25T18:08:37.818 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.816+0000 7f80cbfff640 1 -- 172.21.15.66:0/2122920969 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4001581441 0 0) 0x7f80d40af940 con 0x7f80d40b4d60 2024-03-25T18:08:37.818 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.816+0000 7f80cbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:08:37.818 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.816+0000 7f80cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:37.818 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.816+0000 7f80cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:37.818 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.816+0000 7f80cbfff640 10 monclient(hunting): my global_id is 4434 2024-03-25T18:08:37.818 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.816+0000 7f80cbfff640 1 -- 172.21.15.66:0/2122920969 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80d40b0a70 con 0x7f80d40b4d60 2024-03-25T18:08:37.827 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.825+0000 7f80cbfff640 1 -- 172.21.15.66:0/2122920969 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2492210197 0 0) 0x7f80c0003520 con 0x7f80d40b1730 2024-03-25T18:08:37.827 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.825+0000 7f80cbfff640 1 -- 172.21.15.66:0/2122920969 >> v1:172.21.15.66:6790/0 conn(0x7f80d414be90 legacy=0x7f80d414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:37.827 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.825+0000 7f80cbfff640 1 -- 172.21.15.66:0/2122920969 >> v1:172.21.15.66:6789/0 conn(0x7f80d40b4d60 legacy=0x7f80d413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:37.827 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.825+0000 7f80cbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:08:37.827 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.825+0000 7f80cbfff640 1 monclient: found mon.noname-c 2024-03-25T18:08:37.827 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.825+0000 7f80cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:37.827 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.825+0000 7f80cbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:08:37.827 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.825+0000 7f80cbfff640 1 -- 172.21.15.66:0/2122920969 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d40b1030 con 0x7f80d40b1730 2024-03-25T18:08:37.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.825+0000 7f80cbfff640 10 monclient: _finish_auth 0 2024-03-25T18:08:37.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.825+0000 7f80cbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:08:37.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.825+0000 7f80cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:37.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.826+0000 7f80cbfff640 1 -- 172.21.15.66:0/2122920969 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f80c4001690 con 0x7f80d40b1730 2024-03-25T18:08:37.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.826+0000 7f80db715640 5 monclient: authenticate success, global_id 4432 2024-03-25T18:08:37.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.826+0000 7f80db715640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:37.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.826+0000 7f80cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:37.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.826+0000 7f80cbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:37.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.826+0000 7f80cbfff640 10 monclient: dump: 2024-03-25T18:08:37.828 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:37.828 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:37.828 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:37.828 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:37.828 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:37.828 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:37.828 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:37.829 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:37.829 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:37.829 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:37.829 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.826+0000 7f80db715640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:37.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.836+0000 7f80cbfff640 1 -- 172.21.15.66:0/2122920969 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80c40017f0 con 0x7f80d40b1730 2024-03-25T18:08:37.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.836+0000 7f80cbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:37.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.836+0000 7f80cbfff640 1 -- 172.21.15.66:0/2122920969 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f80c4001c00 con 0x7f80d40b1730 2024-03-25T18:08:37.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.836+0000 7f80db715640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:37.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.836+0000 7f80db715640 4 set_mon_vals no callback set 2024-03-25T18:08:37.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.836+0000 7f80cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:37.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.836+0000 7f80cbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:37.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.836+0000 7f80cbfff640 10 monclient: dump: 2024-03-25T18:08:37.837 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:37.838 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:37.838 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:37.838 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:37.838 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:37.838 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:37.838 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:37.838 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:37.838 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:37.838 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:37.839 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.836+0000 7f80db715640 10 monclient: shutdown 2024-03-25T18:08:37.839 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.836+0000 7f80db715640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:37.839 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.836+0000 7f80db715640 1 -- 172.21.15.66:0/2122920969 >> v1:172.21.15.193:6789/0 conn(0x7f80d40b1730 legacy=0x7f80d40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:37.839 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.837+0000 7f80db715640 1 -- 172.21.15.66:0/2122920969 shutdown_connections 2024-03-25T18:08:37.839 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.837+0000 7f80db715640 1 -- 172.21.15.66:0/2122920969 >> 172.21.15.66:0/2122920969 conn(0x7f80d40617a0 msgr2=0x7f80d4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:37.839 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.837+0000 7f80db715640 1 -- 172.21.15.66:0/2122920969 shutdown_connections 2024-03-25T18:08:37.839 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.837+0000 7f80db715640 1 -- 172.21.15.66:0/2122920969 wait complete. 2024-03-25T18:08:37.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.838+0000 7f80db715640 10 monclient: build_initial_monmap 2024-03-25T18:08:37.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.838+0000 7f80db715640 10 monclient: monmap: 2024-03-25T18:08:37.840 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:08:37.840 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:37.840 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:08:37.840 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:08:37.840 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:37.841 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:08:37.841 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:08:37.841 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:08:37.841 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:08:37.841 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:37.841 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.838+0000 7f80db715640 1 librados: starting msgr at 2024-03-25T18:08:37.841 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.838+0000 7f80db715640 1 librados: starting objecter 2024-03-25T18:08:37.841 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.838+0000 7f80db715640 1 Processor -- start 2024-03-25T18:08:37.841 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.839+0000 7f80db715640 1 -- start start 2024-03-25T18:08:37.841 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.839+0000 7f80db715640 1 librados: setting wanted keys 2024-03-25T18:08:37.841 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.839+0000 7f80db715640 1 librados: calling monclient init 2024-03-25T18:08:37.841 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.839+0000 7f80db715640 10 monclient: init 2024-03-25T18:08:37.842 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.840+0000 7f80db715640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:37.842 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.840+0000 7f80db715640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:08:37.843 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.840+0000 7f80db715640 10 monclient(hunting): picked mon.noname-b con 0x7f80d40b1730 addr v1:172.21.15.193:6789/0 2024-03-25T18:08:37.843 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.840+0000 7f80db715640 10 monclient(hunting): picked mon.noname-c con 0x7f80d40b4d60 addr v1:172.21.15.66:6790/0 2024-03-25T18:08:37.843 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.840+0000 7f80db715640 10 monclient(hunting): picked mon.noname-a con 0x7f80d414be90 addr v1:172.21.15.66:6789/0 2024-03-25T18:08:37.843 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.840+0000 7f80db715640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d4141790 con 0x7f80d414be90 2024-03-25T18:08:37.843 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.840+0000 7f80db715640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d4141a00 con 0x7f80d40b1730 2024-03-25T18:08:37.843 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.840+0000 7f80db715640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80d41d7a90 con 0x7f80d40b4d60 2024-03-25T18:08:37.843 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.840+0000 7f80db715640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:37.843 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.840+0000 7f80db715640 10 monclient(hunting): authenticate will time out at 7266.900372s 2024-03-25T18:08:37.847 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.846+0000 7f80d8cb0640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f80d40b4d60 0x7f80d4142a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:49850/0 (socket says 172.21.15.66:49850) 2024-03-25T18:08:37.847 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.846+0000 7f80d8cb0640 1 -- 172.21.15.66:0/3096150266 learned_addr learned my addr 172.21.15.66:0/3096150266 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:08:37.848 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.847+0000 7f80c9ffb640 10 client.?.objecter ms_handle_connect 0x7f80d40b4d60 2024-03-25T18:08:37.848 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.847+0000 7f80c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:37.854 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.852+0000 7f80c9ffb640 10 client.?.objecter ms_handle_connect 0x7f80d40b1730 2024-03-25T18:08:37.854 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.852+0000 7f80c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:37.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.855+0000 7f80c9ffb640 10 client.?.objecter ms_handle_connect 0x7f80d414be90 2024-03-25T18:08:37.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.855+0000 7f80c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:37.860 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.859+0000 7f80c9ffb640 1 -- 172.21.15.66:0/3096150266 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1488109106 0 0) 0x7f80d4141a00 con 0x7f80d40b1730 2024-03-25T18:08:37.860 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.859+0000 7f80c9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:08:37.860 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.859+0000 7f80c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:37.861 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.859+0000 7f80c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:37.861 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.859+0000 7f80c9ffb640 10 monclient(hunting): my global_id is 4435 2024-03-25T18:08:37.861 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.859+0000 7f80c9ffb640 1 -- 172.21.15.66:0/3096150266 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80b8004eb0 con 0x7f80d40b1730 2024-03-25T18:08:37.865 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.864+0000 7f80c9ffb640 1 -- 172.21.15.66:0/3096150266 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129926265 0 0) 0x7f80d41d7a90 con 0x7f80d40b4d60 2024-03-25T18:08:37.866 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.864+0000 7f80c9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:08:37.866 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.864+0000 7f80c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:37.866 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.864+0000 7f80c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:37.866 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.864+0000 7f80c9ffb640 10 monclient(hunting): my global_id is 4436 2024-03-25T18:08:37.866 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.864+0000 7f80c9ffb640 1 -- 172.21.15.66:0/3096150266 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80d4141a00 con 0x7f80d40b4d60 2024-03-25T18:08:37.878 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.877+0000 7f80c9ffb640 1 -- 172.21.15.66:0/3096150266 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368295426 0 0) 0x7f80d4141790 con 0x7f80d414be90 2024-03-25T18:08:37.878 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.877+0000 7f80c9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:08:37.879 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.877+0000 7f80c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:37.879 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.877+0000 7f80c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:37.879 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.877+0000 7f80c9ffb640 10 monclient(hunting): my global_id is 4437 2024-03-25T18:08:37.879 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.877+0000 7f80c9ffb640 1 -- 172.21.15.66:0/3096150266 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80d41d7a90 con 0x7f80d414be90 2024-03-25T18:08:37.881 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.879+0000 7f80c9ffb640 1 -- 172.21.15.66:0/3096150266 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2505664240 0 0) 0x7f80d4141a00 con 0x7f80d40b4d60 2024-03-25T18:08:37.881 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.879+0000 7f80c9ffb640 1 -- 172.21.15.66:0/3096150266 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80d4141790 con 0x7f80d40b4d60 2024-03-25T18:08:37.881 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.879+0000 7f80c9ffb640 1 -- 172.21.15.66:0/3096150266 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f80c4001b40 con 0x7f80d40b4d60 2024-03-25T18:08:37.881 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.879+0000 7f80c9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:37.881 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.879+0000 7f80c9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:37.881 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.879+0000 7f80c9ffb640 10 monclient(hunting): dump: 2024-03-25T18:08:37.881 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:37.881 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:37.881 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:37.881 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:37.881 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:37.881 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:37.882 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:37.882 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:37.882 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:37.882 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:37.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.879+0000 7f80c9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:37.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.881+0000 7f80c9ffb640 1 -- 172.21.15.66:0/3096150266 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2644932122 0 0) 0x7f80b8004eb0 con 0x7f80d40b1730 2024-03-25T18:08:37.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.881+0000 7f80c9ffb640 1 -- 172.21.15.66:0/3096150266 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80d4141a00 con 0x7f80d40b1730 2024-03-25T18:08:37.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.881+0000 7f80c9ffb640 1 -- 172.21.15.66:0/3096150266 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f80d0001930 con 0x7f80d40b1730 2024-03-25T18:08:37.883 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.881+0000 7f80c9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:37.883 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.881+0000 7f80c9ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:37.883 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.881+0000 7f80c9ffb640 10 monclient(hunting): dump: 2024-03-25T18:08:37.883 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:37.883 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:37.883 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:37.883 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:37.883 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:37.883 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:37.883 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:37.883 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:37.883 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:37.883 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:37.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.890+0000 7f80c9ffb640 1 -- 172.21.15.66:0/3096150266 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1610645389 0 0) 0x7f80d4141a00 con 0x7f80d40b1730 2024-03-25T18:08:37.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.890+0000 7f80c9ffb640 1 -- 172.21.15.66:0/3096150266 >> v1:172.21.15.66:6790/0 conn(0x7f80d40b4d60 legacy=0x7f80d4142a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:37.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.890+0000 7f80c9ffb640 1 -- 172.21.15.66:0/3096150266 >> v1:172.21.15.66:6789/0 conn(0x7f80d414be90 legacy=0x7f80d41d7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:37.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.890+0000 7f80c9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:08:37.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.890+0000 7f80c9ffb640 1 monclient: found mon.b 2024-03-25T18:08:37.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.890+0000 7f80c9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:37.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.890+0000 7f80c9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:08:37.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.890+0000 7f80c9ffb640 1 -- 172.21.15.66:0/3096150266 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d4143140 con 0x7f80d40b1730 2024-03-25T18:08:37.892 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.890+0000 7f80c9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:08:37.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.890+0000 7f80c9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:08:37.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.890+0000 7f80c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:37.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.890+0000 7f80db715640 5 monclient: authenticate success, global_id 4435 2024-03-25T18:08:37.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.890+0000 7f80db715640 10 monclient: _renew_subs 2024-03-25T18:08:37.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.890+0000 7f80db715640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:08:37.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.890+0000 7f80db715640 1 -- 172.21.15.66:0/3096150266 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80d41d86c0 con 0x7f80d40b1730 2024-03-25T18:08:37.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.891+0000 7f80db715640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:37.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.891+0000 7f80db715640 10 monclient: _renew_subs 2024-03-25T18:08:37.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.891+0000 7f80db715640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:08:37.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.891+0000 7f80db715640 1 -- 172.21.15.66:0/3096150266 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f80d41422e0 con 0x7f80d40b1730 2024-03-25T18:08:37.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.891+0000 7f80db715640 1 librados: init done 2024-03-25T18:08:37.894 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.891+0000 7f80db715640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:37.894 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.891+0000 7f80db715640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:37.894 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.891+0000 7f80db715640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:08:37.894 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:37.891+0000 7f80db715640 1 -- 172.21.15.66:0/3096150266 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80d41d7c20 con 0x7f80d40b1730 2024-03-25T18:08:38.171 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.170+0000 7f80c9ffb640 1 -- 172.21.15.66:0/3096150266 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80d0003610 con 0x7f80d40b1730 2024-03-25T18:08:38.172 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.170+0000 7f80c9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:38.172 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.170+0000 7f80c9ffb640 1 -- 172.21.15.66:0/3096150266 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f80d0003920 con 0x7f80d40b1730 2024-03-25T18:08:38.172 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.170+0000 7f80c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:38.172 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.170+0000 7f80c9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:38.172 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.170+0000 7f80c9ffb640 10 monclient: dump: 2024-03-25T18:08:38.172 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:38.172 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:38.172 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:38.172 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:38.172 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:38.172 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:38.172 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:38.172 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:38.172 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:38.172 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:38.173 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.170+0000 7f80cb7fe640 4 set_mon_vals no callback set 2024-03-25T18:08:38.173 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.171+0000 7f80c9ffb640 1 -- 172.21.15.66:0/3096150266 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f80d0019930 con 0x7f80d40b1730 2024-03-25T18:08:38.173 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.172+0000 7f80d8cb0640 10 monclient: get_auth_request con 0x7f80b806df70 auth_method 0 2024-03-25T18:08:38.173 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.172+0000 7f80c9ffb640 10 client.4435.objecter ms_handle_connect 0x7f80b806df70 2024-03-25T18:08:38.175 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.174+0000 7f80c9ffb640 1 -- 172.21.15.66:0/3096150266 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f80d0053650 con 0x7f80d40b1730 2024-03-25T18:08:38.175 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.174+0000 7f80c9ffb640 10 client.4435.objecter ms_dispatch 0x7f80d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:38.175 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.174+0000 7f80c9ffb640 3 client.4435.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:38.175 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.174+0000 7f80c9ffb640 3 client.4435.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:38.175 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.174+0000 7f80c9ffb640 20 client.4435.objecter dump_active .. 0 homeless 2024-03-25T18:08:38.178 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.176+0000 7f80c9ffb640 1 -- 172.21.15.66:0/3096150266 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f80d004ef90 con 0x7f80d40b1730 2024-03-25T18:08:38.178 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.176+0000 7f80c9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:38.178 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.176+0000 7f80c9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:38.429 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.427+0000 7f80db715640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:38.429 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.427+0000 7f80db715640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:38.429 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.427+0000 7f80db715640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:08:38.429 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.427+0000 7f80db715640 1 -- 172.21.15.66:0/3096150266 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80d40627b0 con 0x7f80d40b1730 2024-03-25T18:08:38.455 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.453+0000 7f80c9ffb640 1 -- 172.21.15.66:0/3096150266 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f80d0051090 con 0x7f80d40b1730 2024-03-25T18:08:38.455 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.454+0000 7f80c9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:38.455 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.454+0000 7f80c9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:38.456 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:08:38.456 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:38.458 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.456+0000 7f80db715640 10 librados: watch_flush enter 2024-03-25T18:08:38.459 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.456+0000 7f80db715640 10 librados: watch_flush exit 2024-03-25T18:08:38.459 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.456+0000 7f80db715640 20 client.4435.objecter shutdown clearing up homeless session... 2024-03-25T18:08:38.459 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.456+0000 7f80db715640 10 client.4435.objecter successfully canceled tick 2024-03-25T18:08:38.459 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.456+0000 7f80db715640 1 -- 172.21.15.66:0/3096150266 >> v1:172.21.15.193:6800/2275676803 conn(0x7f80b806df70 legacy=0x7f80b8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:38.459 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.456+0000 7f80db715640 10 monclient: shutdown 2024-03-25T18:08:38.459 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.456+0000 7f80db715640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:38.459 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.456+0000 7f80db715640 1 -- 172.21.15.66:0/3096150266 >> v1:172.21.15.193:6789/0 conn(0x7f80d40b1730 legacy=0x7f80d413b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:38.459 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.456+0000 7f80db715640 1 -- 172.21.15.66:0/3096150266 shutdown_connections 2024-03-25T18:08:38.459 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.456+0000 7f80db715640 1 -- 172.21.15.66:0/3096150266 >> 172.21.15.66:0/3096150266 conn(0x7f80d40617a0 msgr2=0x7f80d40b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:38.459 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.457+0000 7f80db715640 1 -- 172.21.15.66:0/3096150266 shutdown_connections 2024-03-25T18:08:38.459 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.457+0000 7f80db715640 1 -- 172.21.15.66:0/3096150266 wait complete. 2024-03-25T18:08:38.459 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:38.457+0000 7f80db715640 1 librados: shutdown 2024-03-25T18:08:38.470 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:08:45.277 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:45.392 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.391+0000 7f8174b02640 10 monclient: get_monmap_and_config 2024-03-25T18:08:45.392 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.391+0000 7f8174b02640 10 monclient: build_initial_monmap 2024-03-25T18:08:45.392 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.391+0000 7f8174b02640 10 monclient: monmap: 2024-03-25T18:08:45.392 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:08:45.392 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:45.392 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:08:45.392576+0000 2024-03-25T18:08:45.392 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:08:45.392576+0000 2024-03-25T18:08:45.393 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:45.393 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:45.393 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:08:45.393 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:08:45.393 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:08:45.393 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:45.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.391+0000 7f8174b02640 1 Processor -- start 2024-03-25T18:08:45.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.391+0000 7f8174b02640 1 -- start start 2024-03-25T18:08:45.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.391+0000 7f8174b02640 10 monclient: init 2024-03-25T18:08:45.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.392+0000 7f8174b02640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:45.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.392+0000 7f8174b02640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:08:45.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.392+0000 7f8174b02640 10 monclient(hunting): picked mon.noname-c con 0x7f81700a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:08:45.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.392+0000 7f8174b02640 10 monclient(hunting): picked mon.noname-b con 0x7f817013f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:08:45.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.392+0000 7f8174b02640 10 monclient(hunting): picked mon.noname-a con 0x7f81701430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:08:45.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.392+0000 7f8174b02640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8170142520 con 0x7f81701430c0 2024-03-25T18:08:45.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.392+0000 7f8174b02640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81700b28a0 con 0x7f81700a98f0 2024-03-25T18:08:45.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.392+0000 7f8174b02640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81700b2a30 con 0x7f817013f710 2024-03-25T18:08:45.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.392+0000 7f8174b02640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:45.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.392+0000 7f8174b02640 10 monclient(hunting): authenticate will time out at 7274.452727s 2024-03-25T18:08:45.429 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.427+0000 7f816e59c640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f81700a98f0 0x7f81700a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:37950/0 (socket says 172.21.15.66:37950) 2024-03-25T18:08:45.429 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.427+0000 7f816e59c640 1 -- 172.21.15.66:0/1680694001 learned_addr learned my addr 172.21.15.66:0/1680694001 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:08:45.466 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.464+0000 7f816d59a640 1 -- 172.21.15.66:0/1680694001 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3802622379 0 0) 0x7f81700b28a0 con 0x7f81700a98f0 2024-03-25T18:08:45.466 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.464+0000 7f816d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:08:45.466 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.464+0000 7f816d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:45.466 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.464+0000 7f816d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:45.466 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.464+0000 7f816d59a640 10 monclient(hunting): my global_id is 4438 2024-03-25T18:08:45.466 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.464+0000 7f816d59a640 1 -- 172.21.15.66:0/1680694001 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8158003520 con 0x7f81700a98f0 2024-03-25T18:08:45.483 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.482+0000 7f816d59a640 1 -- 172.21.15.66:0/1680694001 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4133422896 0 0) 0x7f81700b2a30 con 0x7f817013f710 2024-03-25T18:08:45.483 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.482+0000 7f816d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:08:45.483 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.482+0000 7f816d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:45.483 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.482+0000 7f816d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:45.483 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.482+0000 7f816d59a640 10 monclient(hunting): my global_id is 4439 2024-03-25T18:08:45.483 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.482+0000 7f816d59a640 1 -- 172.21.15.66:0/1680694001 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81700b28a0 con 0x7f817013f710 2024-03-25T18:08:45.485 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.483+0000 7f816d59a640 1 -- 172.21.15.66:0/1680694001 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1176571906 0 0) 0x7f8158003520 con 0x7f81700a98f0 2024-03-25T18:08:45.485 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.483+0000 7f816d59a640 1 -- 172.21.15.66:0/1680694001 >> v1:172.21.15.66:6790/0 conn(0x7f817013f710 legacy=0x7f81701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:45.485 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.483+0000 7f816d59a640 1 -- 172.21.15.66:0/1680694001 >> v1:172.21.15.66:6789/0 conn(0x7f81701430c0 legacy=0x7f817014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:45.485 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.483+0000 7f816d59a640 10 monclient: _finish_hunting 0 2024-03-25T18:08:45.485 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.483+0000 7f816d59a640 1 monclient: found mon.noname-c 2024-03-25T18:08:45.485 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.483+0000 7f816d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:45.485 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.483+0000 7f816d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:08:45.485 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.483+0000 7f816d59a640 1 -- 172.21.15.66:0/1680694001 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81700b44c0 con 0x7f81700a98f0 2024-03-25T18:08:45.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.483+0000 7f816d59a640 10 monclient: _finish_auth 0 2024-03-25T18:08:45.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.483+0000 7f816d59a640 10 monclient: _check_auth_tickets 2024-03-25T18:08:45.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.483+0000 7f816d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:45.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.483+0000 7f816d59a640 1 -- 172.21.15.66:0/1680694001 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8160001690 con 0x7f81700a98f0 2024-03-25T18:08:45.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.483+0000 7f816d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:45.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.483+0000 7f816d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:45.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.483+0000 7f816d59a640 10 monclient: dump: 2024-03-25T18:08:45.486 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:45.486 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:45.486 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:45.486 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:45.486 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:45.486 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:45.487 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:45.487 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:45.487 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:45.487 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:45.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.483+0000 7f8174b02640 5 monclient: authenticate success, global_id 4438 2024-03-25T18:08:45.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.483+0000 7f8174b02640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:45.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.496+0000 7f816d59a640 1 -- 172.21.15.66:0/1680694001 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8160001870 con 0x7f81700a98f0 2024-03-25T18:08:45.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.496+0000 7f816d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:45.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.496+0000 7f816d59a640 1 -- 172.21.15.66:0/1680694001 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8160001c80 con 0x7f81700a98f0 2024-03-25T18:08:45.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.496+0000 7f816d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:45.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.496+0000 7f816d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:45.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.496+0000 7f816d59a640 10 monclient: dump: 2024-03-25T18:08:45.498 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:45.498 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:45.498 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:45.498 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:45.499 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:45.499 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:45.499 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:45.499 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:45.499 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:45.499 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:45.499 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.496+0000 7f8174b02640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:45.499 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.496+0000 7f8174b02640 4 set_mon_vals no callback set 2024-03-25T18:08:45.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.497+0000 7f8174b02640 10 monclient: shutdown 2024-03-25T18:08:45.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.497+0000 7f8174b02640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:45.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.497+0000 7f8174b02640 1 -- 172.21.15.66:0/1680694001 >> v1:172.21.15.193:6789/0 conn(0x7f81700a98f0 legacy=0x7f81700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:45.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.497+0000 7f8174b02640 1 -- 172.21.15.66:0/1680694001 shutdown_connections 2024-03-25T18:08:45.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.497+0000 7f8174b02640 1 -- 172.21.15.66:0/1680694001 >> 172.21.15.66:0/1680694001 conn(0x7f81700b7ea0 msgr2=0x7f81700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:45.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.497+0000 7f8174b02640 1 -- 172.21.15.66:0/1680694001 shutdown_connections 2024-03-25T18:08:45.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.497+0000 7f8174b02640 1 -- 172.21.15.66:0/1680694001 wait complete. 2024-03-25T18:08:45.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.498+0000 7f8174b02640 10 monclient: build_initial_monmap 2024-03-25T18:08:45.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.498+0000 7f8174b02640 10 monclient: monmap: 2024-03-25T18:08:45.501 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:08:45.501 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:45.501 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:08:45.501 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:08:45.501 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:45.501 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:08:45.501 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:08:45.501 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:08:45.501 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:08:45.501 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:45.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.499+0000 7f8174b02640 1 librados: starting msgr at 2024-03-25T18:08:45.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.499+0000 7f8174b02640 1 librados: starting objecter 2024-03-25T18:08:45.502 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.499+0000 7f8174b02640 1 Processor -- start 2024-03-25T18:08:45.502 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.499+0000 7f8174b02640 1 -- start start 2024-03-25T18:08:45.502 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.499+0000 7f8174b02640 1 librados: setting wanted keys 2024-03-25T18:08:45.502 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.499+0000 7f8174b02640 1 librados: calling monclient init 2024-03-25T18:08:45.502 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.499+0000 7f8174b02640 10 monclient: init 2024-03-25T18:08:45.503 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.500+0000 7f8174b02640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:45.503 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.500+0000 7f8174b02640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:08:45.503 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.500+0000 7f8174b02640 10 monclient(hunting): picked mon.noname-a con 0x7f81700a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:08:45.503 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.500+0000 7f8174b02640 10 monclient(hunting): picked mon.noname-c con 0x7f817013f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:08:45.503 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.500+0000 7f8174b02640 10 monclient(hunting): picked mon.noname-b con 0x7f81701430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:08:45.503 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.500+0000 7f8174b02640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8170135730 con 0x7f81700a98f0 2024-03-25T18:08:45.503 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.500+0000 7f8174b02640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81701358c0 con 0x7f81701430c0 2024-03-25T18:08:45.503 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.500+0000 7f8174b02640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8170135a50 con 0x7f817013f710 2024-03-25T18:08:45.503 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.500+0000 7f8174b02640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:45.503 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.500+0000 7f8174b02640 10 monclient(hunting): authenticate will time out at 7274.560711s 2024-03-25T18:08:45.506 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.505+0000 7f816e59c640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f81700a98f0 0x7f81701dae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:36648/0 (socket says 172.21.15.66:36648) 2024-03-25T18:08:45.506 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.505+0000 7f816e59c640 1 -- 172.21.15.66:0/1343581419 learned_addr learned my addr 172.21.15.66:0/1343581419 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:08:45.506 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.505+0000 7f815effd640 10 client.?.objecter ms_handle_connect 0x7f81700a98f0 2024-03-25T18:08:45.506 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.505+0000 7f815effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:45.509 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.508+0000 7f815effd640 10 client.?.objecter ms_handle_connect 0x7f817013f710 2024-03-25T18:08:45.509 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.508+0000 7f815effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:45.509 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.508+0000 7f815effd640 10 client.?.objecter ms_handle_connect 0x7f81701430c0 2024-03-25T18:08:45.509 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.508+0000 7f815effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:45.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.519+0000 7f815effd640 1 -- 172.21.15.66:0/1343581419 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57559950 0 0) 0x7f8170135a50 con 0x7f817013f710 2024-03-25T18:08:45.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.519+0000 7f815effd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:45.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.519+0000 7f815effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:45.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.519+0000 7f815effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:45.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.519+0000 7f815effd640 10 monclient(hunting): my global_id is 4442 2024-03-25T18:08:45.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.519+0000 7f815effd640 1 -- 172.21.15.66:0/1343581419 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8150004eb0 con 0x7f817013f710 2024-03-25T18:08:45.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.528+0000 7f815effd640 1 -- 172.21.15.66:0/1343581419 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 256837914 0 0) 0x7f81701358c0 con 0x7f81701430c0 2024-03-25T18:08:45.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.528+0000 7f815effd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:45.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.528+0000 7f815effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:45.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.528+0000 7f815effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:45.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.528+0000 7f815effd640 10 monclient(hunting): my global_id is 4441 2024-03-25T18:08:45.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.528+0000 7f815effd640 1 -- 172.21.15.66:0/1343581419 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8170135a50 con 0x7f81701430c0 2024-03-25T18:08:45.534 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.532+0000 7f815effd640 1 -- 172.21.15.66:0/1343581419 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723971742 0 0) 0x7f8170135730 con 0x7f81700a98f0 2024-03-25T18:08:45.534 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.532+0000 7f815effd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:45.534 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.532+0000 7f815effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:45.534 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.532+0000 7f815effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:45.534 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.532+0000 7f815effd640 10 monclient(hunting): my global_id is 4443 2024-03-25T18:08:45.534 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.532+0000 7f815effd640 1 -- 172.21.15.66:0/1343581419 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81701358c0 con 0x7f81700a98f0 2024-03-25T18:08:45.536 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.535+0000 7f815effd640 1 -- 172.21.15.66:0/1343581419 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 574967565 0 0) 0x7f8150004eb0 con 0x7f817013f710 2024-03-25T18:08:45.536 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.535+0000 7f815effd640 1 -- 172.21.15.66:0/1343581419 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8170135730 con 0x7f817013f710 2024-03-25T18:08:45.537 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.535+0000 7f815effd640 1 -- 172.21.15.66:0/1343581419 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8164001930 con 0x7f817013f710 2024-03-25T18:08:45.537 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.535+0000 7f815effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:45.537 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.535+0000 7f815effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:45.537 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.535+0000 7f815effd640 10 monclient(hunting): dump: 2024-03-25T18:08:45.537 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:45.537 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:45.537 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:45.537 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:45.537 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:45.537 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:45.537 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:45.537 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:45.538 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:45.538 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:45.538 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.535+0000 7f815effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:45.546 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.544+0000 7f815effd640 1 -- 172.21.15.66:0/1343581419 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1598938156 0 0) 0x7f8170135a50 con 0x7f81701430c0 2024-03-25T18:08:45.546 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.544+0000 7f815effd640 1 -- 172.21.15.66:0/1343581419 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8150004eb0 con 0x7f81701430c0 2024-03-25T18:08:45.546 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.544+0000 7f815effd640 1 -- 172.21.15.66:0/1343581419 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8168001b00 con 0x7f81701430c0 2024-03-25T18:08:45.546 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.544+0000 7f815effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:45.546 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.544+0000 7f815effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:45.546 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.544+0000 7f815effd640 10 monclient(hunting): dump: 2024-03-25T18:08:45.546 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:45.546 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:45.546 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:45.546 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:45.546 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:45.546 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:45.547 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:45.547 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:45.547 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:45.547 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:45.553 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.551+0000 7f815effd640 1 -- 172.21.15.66:0/1343581419 <== mon.2 v1:172.21.15.66:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 298647036 0 0) 0x7f8170135730 con 0x7f817013f710 2024-03-25T18:08:45.553 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.551+0000 7f815effd640 1 -- 172.21.15.66:0/1343581419 >> v1:172.21.15.193:6789/0 conn(0x7f81701430c0 legacy=0x7f81701d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:45.553 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.551+0000 7f815effd640 1 -- 172.21.15.66:0/1343581419 >> v1:172.21.15.66:6789/0 conn(0x7f81700a98f0 legacy=0x7f81701dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:45.553 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.551+0000 7f815effd640 10 monclient: _finish_hunting 0 2024-03-25T18:08:45.553 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.551+0000 7f815effd640 1 monclient: found mon.c 2024-03-25T18:08:45.553 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.551+0000 7f815effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:45.553 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.551+0000 7f815effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:45.553 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.551+0000 7f815effd640 1 -- 172.21.15.66:0/1343581419 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81701d5000 con 0x7f817013f710 2024-03-25T18:08:45.553 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.551+0000 7f815effd640 10 monclient: _finish_auth 0 2024-03-25T18:08:45.553 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.551+0000 7f815effd640 10 monclient: _check_auth_tickets 2024-03-25T18:08:45.554 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.551+0000 7f815effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:45.554 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.551+0000 7f8174b02640 5 monclient: authenticate success, global_id 4442 2024-03-25T18:08:45.554 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.551+0000 7f8174b02640 10 monclient: _renew_subs 2024-03-25T18:08:45.554 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.551+0000 7f8174b02640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:45.554 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.551+0000 7f8174b02640 1 -- 172.21.15.66:0/1343581419 --> v1:172.21.15.66:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81701d52b0 con 0x7f817013f710 2024-03-25T18:08:45.554 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.551+0000 7f8174b02640 10 client.4442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:45.554 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.551+0000 7f8174b02640 10 monclient: _renew_subs 2024-03-25T18:08:45.554 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.551+0000 7f8174b02640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:45.554 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.551+0000 7f8174b02640 1 -- 172.21.15.66:0/1343581419 --> v1:172.21.15.66:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f817013c900 con 0x7f817013f710 2024-03-25T18:08:45.554 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.551+0000 7f8174b02640 1 librados: init done 2024-03-25T18:08:45.555 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.552+0000 7f8174b02640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:45.555 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.552+0000 7f8174b02640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:45.555 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.552+0000 7f8174b02640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:45.555 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.552+0000 7f8174b02640 1 -- 172.21.15.66:0/1343581419 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81701371c0 con 0x7f817013f710 2024-03-25T18:08:45.556 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.555+0000 7f815effd640 1 -- 172.21.15.66:0/1343581419 <== mon.2 v1:172.21.15.66:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8164003610 con 0x7f817013f710 2024-03-25T18:08:45.556 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.555+0000 7f815effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:45.557 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.555+0000 7f816cd99640 4 set_mon_vals no callback set 2024-03-25T18:08:45.863 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.861+0000 7f815effd640 1 -- 172.21.15.66:0/1343581419 <== mon.2 v1:172.21.15.66:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8164003920 con 0x7f817013f710 2024-03-25T18:08:45.863 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.861+0000 7f815effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:45.863 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.861+0000 7f815effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:45.863 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.861+0000 7f815effd640 10 monclient: dump: 2024-03-25T18:08:45.863 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:45.863 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:45.863 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:45.863 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:45.863 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:45.863 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:45.863 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:45.864 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:45.864 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:45.864 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:45.865 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.862+0000 7f815effd640 1 -- 172.21.15.66:0/1343581419 <== mon.2 v1:172.21.15.66:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f8164019970 con 0x7f817013f710 2024-03-25T18:08:45.866 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.863+0000 7f816e59c640 10 monclient: get_auth_request con 0x7f815006df70 auth_method 0 2024-03-25T18:08:45.866 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.863+0000 7f815effd640 10 client.4442.objecter ms_handle_connect 0x7f815006df70 2024-03-25T18:08:45.869 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.867+0000 7f815effd640 1 -- 172.21.15.66:0/1343581419 <== mon.2 v1:172.21.15.66:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f81640536c0 con 0x7f817013f710 2024-03-25T18:08:45.869 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.867+0000 7f815effd640 10 client.4442.objecter ms_dispatch 0x7f8170000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:45.870 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.867+0000 7f815effd640 3 client.4442.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:45.870 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.867+0000 7f815effd640 3 client.4442.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:45.870 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.868+0000 7f815effd640 20 client.4442.objecter dump_active .. 0 homeless 2024-03-25T18:08:45.870 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.868+0000 7f815effd640 1 -- 172.21.15.66:0/1343581419 <== mon.2 v1:172.21.15.66:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f816404f000 con 0x7f817013f710 2024-03-25T18:08:45.870 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.868+0000 7f815effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:45.870 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:45.868+0000 7f815effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:46.125 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:46.123+0000 7f8174b02640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:46.125 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:46.123+0000 7f8174b02640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:46.125 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:46.123+0000 7f8174b02640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:46.125 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:46.123+0000 7f8174b02640 1 -- 172.21.15.66:0/1343581419 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81700627b0 con 0x7f817013f710 2024-03-25T18:08:46.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:46.147+0000 7f815effd640 1 -- 172.21.15.66:0/1343581419 <== mon.2 v1:172.21.15.66:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f8164051100 con 0x7f817013f710 2024-03-25T18:08:46.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:46.147+0000 7f815effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:46.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:46.147+0000 7f815effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:46.149 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:08:46.149 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:46.152 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:46.150+0000 7f8174b02640 10 librados: watch_flush enter 2024-03-25T18:08:46.152 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:46.150+0000 7f8174b02640 10 librados: watch_flush exit 2024-03-25T18:08:46.152 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:46.150+0000 7f8174b02640 20 client.4442.objecter shutdown clearing up homeless session... 2024-03-25T18:08:46.152 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:46.150+0000 7f8174b02640 10 client.4442.objecter successfully canceled tick 2024-03-25T18:08:46.152 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:46.150+0000 7f8174b02640 1 -- 172.21.15.66:0/1343581419 >> v1:172.21.15.193:6800/2275676803 conn(0x7f815006df70 legacy=0x7f8150070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:46.152 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:46.150+0000 7f8174b02640 10 monclient: shutdown 2024-03-25T18:08:46.152 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:46.150+0000 7f8174b02640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:46.152 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:46.150+0000 7f8174b02640 1 -- 172.21.15.66:0/1343581419 >> v1:172.21.15.66:6790/0 conn(0x7f817013f710 legacy=0x7f81701d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:46.152 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:46.150+0000 7f8174b02640 1 -- 172.21.15.66:0/1343581419 shutdown_connections 2024-03-25T18:08:46.152 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:46.150+0000 7f8174b02640 1 -- 172.21.15.66:0/1343581419 >> 172.21.15.66:0/1343581419 conn(0x7f81700b7ea0 msgr2=0x7f817014ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:46.153 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:46.150+0000 7f8174b02640 1 -- 172.21.15.66:0/1343581419 shutdown_connections 2024-03-25T18:08:46.153 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:46.150+0000 7f8174b02640 1 -- 172.21.15.66:0/1343581419 wait complete. 2024-03-25T18:08:46.153 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:46.150+0000 7f8174b02640 1 librados: shutdown 2024-03-25T18:08:46.164 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:08:49.372 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:05:20.352 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:49.417 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:05:20.375 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:49.418 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:05:20.376 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:49.428 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:05:20.409 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:49.463 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:05:20.421 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:49.463 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:05:20.443 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:49.489 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:05:20.470 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:49.494 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:05:20.475 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:49.513 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:05:20.493 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:52.970 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:53.095 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.093+0000 7f1b39d54640 10 monclient: get_monmap_and_config 2024-03-25T18:08:53.095 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.093+0000 7f1b39d54640 10 monclient: build_initial_monmap 2024-03-25T18:08:53.095 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.093+0000 7f1b39d54640 10 monclient: monmap: 2024-03-25T18:08:53.095 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:08:53.095 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:53.095 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:08:53.095200+0000 2024-03-25T18:08:53.095 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:08:53.095200+0000 2024-03-25T18:08:53.095 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:53.096 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:53.096 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:08:53.096 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:08:53.096 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:08:53.096 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:53.097 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.094+0000 7f1b39d54640 1 Processor -- start 2024-03-25T18:08:53.097 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.094+0000 7f1b39d54640 1 -- start start 2024-03-25T18:08:53.097 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.094+0000 7f1b39d54640 10 monclient: init 2024-03-25T18:08:53.097 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.095+0000 7f1b39d54640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:53.097 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.095+0000 7f1b39d54640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:08:53.097 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.095+0000 7f1b39d54640 10 monclient(hunting): picked mon.noname-c con 0x7f1b340a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:08:53.097 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.095+0000 7f1b39d54640 10 monclient(hunting): picked mon.noname-b con 0x7f1b3413f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:08:53.097 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.095+0000 7f1b39d54640 10 monclient(hunting): picked mon.noname-a con 0x7f1b341430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:08:53.098 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.095+0000 7f1b39d54640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b34142520 con 0x7f1b341430c0 2024-03-25T18:08:53.098 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.095+0000 7f1b39d54640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b340b28a0 con 0x7f1b340a98f0 2024-03-25T18:08:53.098 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.095+0000 7f1b39d54640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b340b2a30 con 0x7f1b3413f710 2024-03-25T18:08:53.098 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.095+0000 7f1b39d54640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:53.098 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.095+0000 7f1b39d54640 10 monclient(hunting): authenticate will time out at 7282.155766s 2024-03-25T18:08:53.111 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.110+0000 7f1b2bfff640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f1b3413f710 0x7f1b341429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:39854/0 (socket says 172.21.15.66:39854) 2024-03-25T18:08:53.111 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.110+0000 7f1b2bfff640 1 -- 172.21.15.66:0/3957281964 learned_addr learned my addr 172.21.15.66:0/3957281964 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:08:53.130 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.129+0000 7f1b32ffd640 1 -- 172.21.15.66:0/3957281964 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2956663049 0 0) 0x7f1b340b2a30 con 0x7f1b3413f710 2024-03-25T18:08:53.130 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.129+0000 7f1b32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:53.130 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.129+0000 7f1b32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:53.130 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.129+0000 7f1b32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:53.130 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.129+0000 7f1b32ffd640 10 monclient(hunting): my global_id is 4445 2024-03-25T18:08:53.130 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.129+0000 7f1b32ffd640 1 -- 172.21.15.66:0/3957281964 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b20003520 con 0x7f1b3413f710 2024-03-25T18:08:53.133 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.132+0000 7f1b32ffd640 1 -- 172.21.15.66:0/3957281964 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 71491093 0 0) 0x7f1b34142520 con 0x7f1b341430c0 2024-03-25T18:08:53.133 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.132+0000 7f1b32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:53.133 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.132+0000 7f1b32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:53.133 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.132+0000 7f1b32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:53.133 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.132+0000 7f1b32ffd640 10 monclient(hunting): my global_id is 4446 2024-03-25T18:08:53.134 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.132+0000 7f1b32ffd640 1 -- 172.21.15.66:0/3957281964 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b340b2a30 con 0x7f1b341430c0 2024-03-25T18:08:53.134 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.133+0000 7f1b32ffd640 1 -- 172.21.15.66:0/3957281964 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3002184164 0 0) 0x7f1b340b28a0 con 0x7f1b340a98f0 2024-03-25T18:08:53.135 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.133+0000 7f1b32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:53.135 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.133+0000 7f1b32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:53.135 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.133+0000 7f1b32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:53.135 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.133+0000 7f1b32ffd640 10 monclient(hunting): my global_id is 4444 2024-03-25T18:08:53.135 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.133+0000 7f1b32ffd640 1 -- 172.21.15.66:0/3957281964 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b34142520 con 0x7f1b340a98f0 2024-03-25T18:08:53.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.143+0000 7f1b32ffd640 1 -- 172.21.15.66:0/3957281964 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 851002050 0 0) 0x7f1b340b2a30 con 0x7f1b341430c0 2024-03-25T18:08:53.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.143+0000 7f1b32ffd640 1 -- 172.21.15.66:0/3957281964 >> v1:172.21.15.66:6790/0 conn(0x7f1b3413f710 legacy=0x7f1b341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:53.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.143+0000 7f1b32ffd640 1 -- 172.21.15.66:0/3957281964 >> v1:172.21.15.193:6789/0 conn(0x7f1b340a98f0 legacy=0x7f1b340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:53.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.143+0000 7f1b32ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:08:53.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.143+0000 7f1b32ffd640 1 monclient: found mon.noname-a 2024-03-25T18:08:53.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.143+0000 7f1b32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:53.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.143+0000 7f1b32ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.66:6789/0 2024-03-25T18:08:53.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.143+0000 7f1b32ffd640 1 -- 172.21.15.66:0/3957281964 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b340b44c0 con 0x7f1b341430c0 2024-03-25T18:08:53.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.143+0000 7f1b32ffd640 10 monclient: _finish_auth 0 2024-03-25T18:08:53.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.143+0000 7f1b32ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:08:53.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.143+0000 7f1b32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:53.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.143+0000 7f1b39d54640 5 monclient: authenticate success, global_id 4446 2024-03-25T18:08:53.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.143+0000 7f1b32ffd640 1 -- 172.21.15.66:0/3957281964 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f1b2c0019f0 con 0x7f1b341430c0 2024-03-25T18:08:53.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.143+0000 7f1b32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:53.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.143+0000 7f1b32ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:08:53.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.143+0000 7f1b32ffd640 10 monclient: dump: 2024-03-25T18:08:53.145 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:53.146 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:53.146 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:53.146 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:53.146 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:53.146 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:53.146 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:53.146 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:53.146 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:53.146 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:53.146 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.143+0000 7f1b39d54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:53.151 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.149+0000 7f1b32ffd640 1 -- 172.21.15.66:0/3957281964 <== mon.0 v1:172.21.15.66:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b2c001b50 con 0x7f1b341430c0 2024-03-25T18:08:53.151 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.149+0000 7f1b32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:53.151 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.149+0000 7f1b32ffd640 1 -- 172.21.15.66:0/3957281964 <== mon.0 v1:172.21.15.66:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f1b2c001f60 con 0x7f1b341430c0 2024-03-25T18:08:53.151 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.149+0000 7f1b39d54640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:53.151 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.149+0000 7f1b39d54640 4 set_mon_vals no callback set 2024-03-25T18:08:53.151 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.149+0000 7f1b32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:53.151 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.149+0000 7f1b32ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:53.151 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.149+0000 7f1b32ffd640 10 monclient: dump: 2024-03-25T18:08:53.152 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:53.152 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:53.152 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:53.152 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:53.152 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:53.152 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:53.152 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:53.152 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:53.152 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:53.152 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:53.153 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.150+0000 7f1b39d54640 10 monclient: shutdown 2024-03-25T18:08:53.153 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.150+0000 7f1b39d54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:53.153 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.150+0000 7f1b39d54640 1 -- 172.21.15.66:0/3957281964 >> v1:172.21.15.66:6789/0 conn(0x7f1b341430c0 legacy=0x7f1b3414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:53.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.150+0000 7f1b39d54640 1 -- 172.21.15.66:0/3957281964 shutdown_connections 2024-03-25T18:08:53.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.150+0000 7f1b39d54640 1 -- 172.21.15.66:0/3957281964 >> 172.21.15.66:0/3957281964 conn(0x7f1b340b7ea0 msgr2=0x7f1b340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:53.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.150+0000 7f1b39d54640 1 -- 172.21.15.66:0/3957281964 shutdown_connections 2024-03-25T18:08:53.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.150+0000 7f1b39d54640 1 -- 172.21.15.66:0/3957281964 wait complete. 2024-03-25T18:08:53.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.151+0000 7f1b39d54640 10 monclient: build_initial_monmap 2024-03-25T18:08:53.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.151+0000 7f1b39d54640 10 monclient: monmap: 2024-03-25T18:08:53.154 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:08:53.154 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:53.154 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:08:53.154 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:08:53.154 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:53.154 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:08:53.154 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:08:53.154 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:08:53.155 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:08:53.155 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:53.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.152+0000 7f1b39d54640 1 librados: starting msgr at 2024-03-25T18:08:53.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.152+0000 7f1b39d54640 1 librados: starting objecter 2024-03-25T18:08:53.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.152+0000 7f1b39d54640 1 Processor -- start 2024-03-25T18:08:53.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.152+0000 7f1b39d54640 1 -- start start 2024-03-25T18:08:53.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.152+0000 7f1b39d54640 1 librados: setting wanted keys 2024-03-25T18:08:53.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.152+0000 7f1b39d54640 1 librados: calling monclient init 2024-03-25T18:08:53.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.152+0000 7f1b39d54640 10 monclient: init 2024-03-25T18:08:53.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.153+0000 7f1b39d54640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:53.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.153+0000 7f1b39d54640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:08:53.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.153+0000 7f1b39d54640 10 monclient(hunting): picked mon.noname-c con 0x7f1b340a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:08:53.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.153+0000 7f1b39d54640 10 monclient(hunting): picked mon.noname-a con 0x7f1b3413f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:08:53.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.153+0000 7f1b39d54640 10 monclient(hunting): picked mon.noname-b con 0x7f1b341430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:08:53.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.153+0000 7f1b39d54640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b34135bf0 con 0x7f1b3413f710 2024-03-25T18:08:53.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.153+0000 7f1b39d54640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b340b6d10 con 0x7f1b341430c0 2024-03-25T18:08:53.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.153+0000 7f1b39d54640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b341317f0 con 0x7f1b340a98f0 2024-03-25T18:08:53.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.153+0000 7f1b39d54640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:53.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.153+0000 7f1b39d54640 10 monclient(hunting): authenticate will time out at 7282.213419s 2024-03-25T18:08:53.164 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.163+0000 7f1b337fe640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f1b340a98f0 0x7f1b341310e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:39894/0 (socket says 172.21.15.66:39894) 2024-03-25T18:08:53.165 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.163+0000 7f1b337fe640 1 -- 172.21.15.66:0/3011615172 learned_addr learned my addr 172.21.15.66:0/3011615172 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:08:53.170 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.168+0000 7f1b30ff9640 10 client.?.objecter ms_handle_connect 0x7f1b340a98f0 2024-03-25T18:08:53.170 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.168+0000 7f1b30ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:53.170 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.169+0000 7f1b30ff9640 10 client.?.objecter ms_handle_connect 0x7f1b341430c0 2024-03-25T18:08:53.171 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.169+0000 7f1b30ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:53.173 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.172+0000 7f1b30ff9640 1 -- 172.21.15.66:0/3011615172 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 234324440 0 0) 0x7f1b341317f0 con 0x7f1b340a98f0 2024-03-25T18:08:53.174 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.172+0000 7f1b30ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:53.174 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.172+0000 7f1b30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:53.174 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.172+0000 7f1b30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:53.174 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.172+0000 7f1b30ff9640 10 monclient(hunting): my global_id is 4448 2024-03-25T18:08:53.174 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.172+0000 7f1b30ff9640 1 -- 172.21.15.66:0/3011615172 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b14004eb0 con 0x7f1b340a98f0 2024-03-25T18:08:53.176 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.175+0000 7f1b30ff9640 10 client.?.objecter ms_handle_connect 0x7f1b3413f710 2024-03-25T18:08:53.176 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.175+0000 7f1b30ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:53.177 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.175+0000 7f1b30ff9640 1 -- 172.21.15.66:0/3011615172 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 224617784 0 0) 0x7f1b340b6d10 con 0x7f1b341430c0 2024-03-25T18:08:53.177 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.175+0000 7f1b30ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:53.177 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.175+0000 7f1b30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:53.177 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.175+0000 7f1b30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:53.177 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.175+0000 7f1b30ff9640 10 monclient(hunting): my global_id is 4447 2024-03-25T18:08:53.177 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.175+0000 7f1b30ff9640 1 -- 172.21.15.66:0/3011615172 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b341317f0 con 0x7f1b341430c0 2024-03-25T18:08:53.182 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.180+0000 7f1b30ff9640 1 -- 172.21.15.66:0/3011615172 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 17518531 0 0) 0x7f1b14004eb0 con 0x7f1b340a98f0 2024-03-25T18:08:53.182 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.180+0000 7f1b30ff9640 1 -- 172.21.15.66:0/3011615172 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b340b6d10 con 0x7f1b340a98f0 2024-03-25T18:08:53.187 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.185+0000 7f1b30ff9640 1 -- 172.21.15.66:0/3011615172 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1136790673 0 0) 0x7f1b34135bf0 con 0x7f1b3413f710 2024-03-25T18:08:53.187 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.185+0000 7f1b30ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:53.187 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.185+0000 7f1b30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:53.187 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.185+0000 7f1b30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:53.187 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.185+0000 7f1b30ff9640 10 monclient(hunting): my global_id is 4449 2024-03-25T18:08:53.187 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.185+0000 7f1b30ff9640 1 -- 172.21.15.66:0/3011615172 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b14004eb0 con 0x7f1b3413f710 2024-03-25T18:08:53.197 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.195+0000 7f1b30ff9640 1 -- 172.21.15.66:0/3011615172 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f1b24001ad0 con 0x7f1b340a98f0 2024-03-25T18:08:53.197 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.195+0000 7f1b30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:53.197 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.195+0000 7f1b30ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:53.197 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.195+0000 7f1b30ff9640 10 monclient(hunting): dump: 2024-03-25T18:08:53.197 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:53.197 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:53.197 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:53.198 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:53.198 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:53.198 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:53.198 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:53.198 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:53.198 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:53.198 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:53.198 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.195+0000 7f1b30ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:53.198 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.195+0000 7f1b30ff9640 1 -- 172.21.15.66:0/3011615172 <== mon.2 v1:172.21.15.66:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 593871355 0 0) 0x7f1b340b6d10 con 0x7f1b340a98f0 2024-03-25T18:08:53.198 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.195+0000 7f1b30ff9640 1 -- 172.21.15.66:0/3011615172 >> v1:172.21.15.193:6789/0 conn(0x7f1b341430c0 legacy=0x7f1b341db370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:53.198 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.195+0000 7f1b30ff9640 1 -- 172.21.15.66:0/3011615172 >> v1:172.21.15.66:6789/0 conn(0x7f1b3413f710 legacy=0x7f1b341dac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:53.198 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.195+0000 7f1b30ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:08:53.198 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.195+0000 7f1b30ff9640 1 monclient: found mon.c 2024-03-25T18:08:53.198 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.195+0000 7f1b30ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:53.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.195+0000 7f1b30ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:53.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.195+0000 7f1b30ff9640 1 -- 172.21.15.66:0/3011615172 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b34131980 con 0x7f1b340a98f0 2024-03-25T18:08:53.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.195+0000 7f1b30ff9640 10 monclient: _finish_auth 0 2024-03-25T18:08:53.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.195+0000 7f1b30ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:08:53.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.195+0000 7f1b30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:53.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.195+0000 7f1b39d54640 5 monclient: authenticate success, global_id 4448 2024-03-25T18:08:53.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.195+0000 7f1b39d54640 10 monclient: _renew_subs 2024-03-25T18:08:53.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.195+0000 7f1b39d54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:53.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.195+0000 7f1b39d54640 1 -- 172.21.15.66:0/3011615172 --> v1:172.21.15.66:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b340b76e0 con 0x7f1b340a98f0 2024-03-25T18:08:53.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.195+0000 7f1b39d54640 10 client.4448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:53.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.195+0000 7f1b39d54640 10 monclient: _renew_subs 2024-03-25T18:08:53.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.195+0000 7f1b39d54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:53.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.195+0000 7f1b39d54640 1 -- 172.21.15.66:0/3011615172 --> v1:172.21.15.66:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b34135270 con 0x7f1b340a98f0 2024-03-25T18:08:53.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.195+0000 7f1b39d54640 1 librados: init done 2024-03-25T18:08:53.200 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.196+0000 7f1b39d54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:53.200 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.196+0000 7f1b39d54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:53.200 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.196+0000 7f1b39d54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:53.200 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.196+0000 7f1b39d54640 1 -- 172.21.15.66:0/3011615172 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b340627b0 con 0x7f1b340a98f0 2024-03-25T18:08:53.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.496+0000 7f1b30ff9640 1 -- 172.21.15.66:0/3011615172 <== mon.2 v1:172.21.15.66:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b24001c30 con 0x7f1b340a98f0 2024-03-25T18:08:53.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.496+0000 7f1b30ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:53.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.496+0000 7f1b327fc640 4 set_mon_vals no callback set 2024-03-25T18:08:53.509 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.508+0000 7f1b30ff9640 1 -- 172.21.15.66:0/3011615172 <== mon.2 v1:172.21.15.66:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f1b24003a60 con 0x7f1b340a98f0 2024-03-25T18:08:53.509 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.508+0000 7f1b30ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:53.509 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.508+0000 7f1b30ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:53.509 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.508+0000 7f1b30ff9640 10 monclient: dump: 2024-03-25T18:08:53.509 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:08:53.509 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:08:53.509 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:53.509 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:08:53.509 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:53.509 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:08:53.510 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:08:53.510 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:08:53.510 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:08:53.510 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:08:53.510 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.508+0000 7f1b30ff9640 1 -- 172.21.15.66:0/3011615172 <== mon.2 v1:172.21.15.66:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f1b24003bc0 con 0x7f1b340a98f0 2024-03-25T18:08:53.510 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.509+0000 7f1b2bfff640 10 monclient: get_auth_request con 0x7f1b1406de60 auth_method 0 2024-03-25T18:08:53.511 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.510+0000 7f1b30ff9640 10 client.4448.objecter ms_handle_connect 0x7f1b1406de60 2024-03-25T18:08:53.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.516+0000 7f1b30ff9640 1 -- 172.21.15.66:0/3011615172 <== mon.2 v1:172.21.15.66:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f1b240548a0 con 0x7f1b340a98f0 2024-03-25T18:08:53.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.516+0000 7f1b30ff9640 10 client.4448.objecter ms_dispatch 0x7f1b34000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:53.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.516+0000 7f1b30ff9640 3 client.4448.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:53.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.516+0000 7f1b30ff9640 3 client.4448.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:53.518 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.516+0000 7f1b30ff9640 20 client.4448.objecter dump_active .. 0 homeless 2024-03-25T18:08:53.518 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.516+0000 7f1b30ff9640 1 -- 172.21.15.66:0/3011615172 <== mon.2 v1:172.21.15.66:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1b240503b0 con 0x7f1b340a98f0 2024-03-25T18:08:53.518 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.516+0000 7f1b30ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:53.518 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.516+0000 7f1b30ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:53.752 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.751+0000 7f1b39d54640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:53.753 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.751+0000 7f1b39d54640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:53.753 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.751+0000 7f1b39d54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:08:53.753 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.751+0000 7f1b39d54640 1 -- 172.21.15.66:0/3011615172 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b3413e100 con 0x7f1b340a98f0 2024-03-25T18:08:53.776 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.775+0000 7f1b30ff9640 1 -- 172.21.15.66:0/3011615172 <== mon.2 v1:172.21.15.66:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f1b240522e0 con 0x7f1b340a98f0 2024-03-25T18:08:53.776 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.775+0000 7f1b30ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:53.777 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.775+0000 7f1b30ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:53.777 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:08:53.777 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:53.780 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.778+0000 7f1b39d54640 10 librados: watch_flush enter 2024-03-25T18:08:53.780 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.778+0000 7f1b39d54640 10 librados: watch_flush exit 2024-03-25T18:08:53.780 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.778+0000 7f1b39d54640 20 client.4448.objecter shutdown clearing up homeless session... 2024-03-25T18:08:53.780 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.778+0000 7f1b39d54640 10 client.4448.objecter successfully canceled tick 2024-03-25T18:08:53.780 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.778+0000 7f1b39d54640 1 -- 172.21.15.66:0/3011615172 >> v1:172.21.15.193:6800/2275676803 conn(0x7f1b1406de60 legacy=0x7f1b14070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:53.780 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.778+0000 7f1b39d54640 10 monclient: shutdown 2024-03-25T18:08:53.781 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.778+0000 7f1b39d54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:53.781 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.778+0000 7f1b39d54640 1 -- 172.21.15.66:0/3011615172 >> v1:172.21.15.66:6790/0 conn(0x7f1b340a98f0 legacy=0x7f1b341310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:53.781 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.778+0000 7f1b39d54640 1 -- 172.21.15.66:0/3011615172 shutdown_connections 2024-03-25T18:08:53.781 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.778+0000 7f1b39d54640 1 -- 172.21.15.66:0/3011615172 >> 172.21.15.66:0/3011615172 conn(0x7f1b340b7ea0 msgr2=0x7f1b34074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:53.781 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.778+0000 7f1b39d54640 1 -- 172.21.15.66:0/3011615172 shutdown_connections 2024-03-25T18:08:53.781 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.778+0000 7f1b39d54640 1 -- 172.21.15.66:0/3011615172 wait complete. 2024-03-25T18:08:53.781 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:08:53.778+0000 7f1b39d54640 1 librados: shutdown 2024-03-25T18:08:53.793 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:00.600 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:00.703 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.702+0000 7f64c67e2640 10 monclient: get_monmap_and_config 2024-03-25T18:09:00.703 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.702+0000 7f64c67e2640 10 monclient: build_initial_monmap 2024-03-25T18:09:00.703 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.702+0000 7f64c67e2640 10 monclient: monmap: 2024-03-25T18:09:00.703 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:09:00.703 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:00.703 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:09:00.703766+0000 2024-03-25T18:09:00.704 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:09:00.703766+0000 2024-03-25T18:09:00.704 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:00.704 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:00.704 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:09:00.704 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:09:00.704 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:09:00.704 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:00.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.703+0000 7f64c67e2640 1 Processor -- start 2024-03-25T18:09:00.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.703+0000 7f64c67e2640 1 -- start start 2024-03-25T18:09:00.705 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.703+0000 7f64c67e2640 10 monclient: init 2024-03-25T18:09:00.706 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.703+0000 7f64c67e2640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:00.706 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.703+0000 7f64c67e2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:09:00.706 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.703+0000 7f64c67e2640 10 monclient(hunting): picked mon.noname-b con 0x7f64c00a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:09:00.706 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.703+0000 7f64c67e2640 10 monclient(hunting): picked mon.noname-c con 0x7f64c013f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:09:00.706 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.704+0000 7f64c67e2640 10 monclient(hunting): picked mon.noname-a con 0x7f64c01430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:09:00.706 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.704+0000 7f64c67e2640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c0142520 con 0x7f64c01430c0 2024-03-25T18:09:00.706 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.704+0000 7f64c67e2640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c00b28a0 con 0x7f64c013f710 2024-03-25T18:09:00.706 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.704+0000 7f64c67e2640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c00b2a30 con 0x7f64c00a98f0 2024-03-25T18:09:00.706 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.704+0000 7f64c67e2640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:00.706 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.704+0000 7f64c67e2640 10 monclient(hunting): authenticate will time out at 7289.764255s 2024-03-25T18:09:00.706 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.704+0000 7f64bf7fe640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f64c013f710 0x7f64c01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:54562/0 (socket says 172.21.15.66:54562) 2024-03-25T18:09:00.707 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.704+0000 7f64bf7fe640 1 -- 172.21.15.66:0/3141042322 learned_addr learned my addr 172.21.15.66:0/3141042322 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:09:00.715 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.713+0000 7f64beffd640 1 -- 172.21.15.66:0/3141042322 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1168192818 0 0) 0x7f64c0142520 con 0x7f64c01430c0 2024-03-25T18:09:00.715 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.713+0000 7f64beffd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:00.715 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.713+0000 7f64beffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:00.715 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.713+0000 7f64beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:00.715 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.713+0000 7f64beffd640 10 monclient(hunting): my global_id is 4452 2024-03-25T18:09:00.715 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.713+0000 7f64beffd640 1 -- 172.21.15.66:0/3141042322 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64a8003520 con 0x7f64c01430c0 2024-03-25T18:09:00.739 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.737+0000 7f64beffd640 1 -- 172.21.15.66:0/3141042322 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2021039989 0 0) 0x7f64a8003520 con 0x7f64c01430c0 2024-03-25T18:09:00.739 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.737+0000 7f64beffd640 1 -- 172.21.15.66:0/3141042322 >> v1:172.21.15.66:6790/0 conn(0x7f64c00a98f0 legacy=0x7f64c00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:09:00.739 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.737+0000 7f64beffd640 1 -- 172.21.15.66:0/3141042322 >> v1:172.21.15.193:6789/0 conn(0x7f64c013f710 legacy=0x7f64c01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:00.739 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.737+0000 7f64beffd640 10 monclient: _finish_hunting 0 2024-03-25T18:09:00.739 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.737+0000 7f64beffd640 1 monclient: found mon.noname-a 2024-03-25T18:09:00.739 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.737+0000 7f64beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:00.739 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.737+0000 7f64beffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.66:6789/0 2024-03-25T18:09:00.739 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.737+0000 7f64beffd640 1 -- 172.21.15.66:0/3141042322 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c00b44c0 con 0x7f64c01430c0 2024-03-25T18:09:00.739 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.737+0000 7f64beffd640 10 monclient: _finish_auth 0 2024-03-25T18:09:00.739 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.737+0000 7f64beffd640 10 monclient: _check_auth_tickets 2024-03-25T18:09:00.739 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.737+0000 7f64beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:00.739 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.737+0000 7f64beffd640 1 -- 172.21.15.66:0/3141042322 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f64b8001a10 con 0x7f64c01430c0 2024-03-25T18:09:00.739 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.737+0000 7f64beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:00.739 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.737+0000 7f64beffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:09:00.740 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.737+0000 7f64beffd640 10 monclient: dump: 2024-03-25T18:09:00.740 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:00.740 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:00.740 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:00.740 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:00.740 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:00.740 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:00.740 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:00.740 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:00.740 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:00.740 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:00.740 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.737+0000 7f64c67e2640 5 monclient: authenticate success, global_id 4452 2024-03-25T18:09:00.740 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.737+0000 7f64c67e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:00.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.756+0000 7f64beffd640 1 -- 172.21.15.66:0/3141042322 <== mon.0 v1:172.21.15.66:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64b8001be0 con 0x7f64c01430c0 2024-03-25T18:09:00.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.756+0000 7f64beffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:00.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.756+0000 7f64beffd640 1 -- 172.21.15.66:0/3141042322 <== mon.0 v1:172.21.15.66:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f64b8001ff0 con 0x7f64c01430c0 2024-03-25T18:09:00.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.756+0000 7f64beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:00.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.756+0000 7f64beffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:00.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.756+0000 7f64beffd640 10 monclient: dump: 2024-03-25T18:09:00.759 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:00.759 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:00.759 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:00.759 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:00.759 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:00.759 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:00.759 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:00.759 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:00.759 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:00.759 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:00.760 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.756+0000 7f64c67e2640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:00.760 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.756+0000 7f64c67e2640 4 set_mon_vals no callback set 2024-03-25T18:09:00.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.757+0000 7f64c67e2640 10 monclient: shutdown 2024-03-25T18:09:00.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.757+0000 7f64c67e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:00.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.757+0000 7f64c67e2640 1 -- 172.21.15.66:0/3141042322 >> v1:172.21.15.66:6789/0 conn(0x7f64c01430c0 legacy=0x7f64c014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:00.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.757+0000 7f64c67e2640 1 -- 172.21.15.66:0/3141042322 shutdown_connections 2024-03-25T18:09:00.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.757+0000 7f64c67e2640 1 -- 172.21.15.66:0/3141042322 >> 172.21.15.66:0/3141042322 conn(0x7f64c00b7ea0 msgr2=0x7f64c00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:00.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.757+0000 7f64c67e2640 1 -- 172.21.15.66:0/3141042322 shutdown_connections 2024-03-25T18:09:00.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.757+0000 7f64c67e2640 1 -- 172.21.15.66:0/3141042322 wait complete. 2024-03-25T18:09:00.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.758+0000 7f64c67e2640 10 monclient: build_initial_monmap 2024-03-25T18:09:00.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.758+0000 7f64c67e2640 10 monclient: monmap: 2024-03-25T18:09:00.763 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:09:00.763 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:00.763 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:09:00.763 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:09:00.763 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:00.763 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:09:00.763 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:09:00.763 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:09:00.763 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:09:00.763 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:00.764 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.758+0000 7f64c67e2640 1 librados: starting msgr at 2024-03-25T18:09:00.764 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.758+0000 7f64c67e2640 1 librados: starting objecter 2024-03-25T18:09:00.764 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.758+0000 7f64c67e2640 1 Processor -- start 2024-03-25T18:09:00.764 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.758+0000 7f64c67e2640 1 -- start start 2024-03-25T18:09:00.764 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.758+0000 7f64c67e2640 1 librados: setting wanted keys 2024-03-25T18:09:00.764 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.758+0000 7f64c67e2640 1 librados: calling monclient init 2024-03-25T18:09:00.764 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.758+0000 7f64c67e2640 10 monclient: init 2024-03-25T18:09:00.764 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.759+0000 7f64c67e2640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:00.764 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.759+0000 7f64c67e2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:09:00.764 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.759+0000 7f64c67e2640 10 monclient(hunting): picked mon.noname-a con 0x7f64c00a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:09:00.764 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.759+0000 7f64c67e2640 10 monclient(hunting): picked mon.noname-c con 0x7f64c013f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:09:00.764 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.759+0000 7f64c67e2640 10 monclient(hunting): picked mon.noname-b con 0x7f64c01430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:09:00.765 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.759+0000 7f64c67e2640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c00abde0 con 0x7f64c00a98f0 2024-03-25T18:09:00.765 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.759+0000 7f64c67e2640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c01d51f0 con 0x7f64c01430c0 2024-03-25T18:09:00.765 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.759+0000 7f64c67e2640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c01d83d0 con 0x7f64c013f710 2024-03-25T18:09:00.765 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.759+0000 7f64c67e2640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:00.765 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.759+0000 7f64c67e2640 10 monclient(hunting): authenticate will time out at 7289.819858s 2024-03-25T18:09:00.765 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.761+0000 7f64c4d7f640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f64c01430c0 0x7f64c01e1dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:54602/0 (socket says 172.21.15.66:54602) 2024-03-25T18:09:00.766 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.761+0000 7f64c4d7f640 1 -- 172.21.15.66:0/3170178378 learned_addr learned my addr 172.21.15.66:0/3170178378 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:09:00.766 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.763+0000 7f64bcff9640 10 client.?.objecter ms_handle_connect 0x7f64c01430c0 2024-03-25T18:09:00.766 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.763+0000 7f64bcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:00.779 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.778+0000 7f64bcff9640 10 client.?.objecter ms_handle_connect 0x7f64c00a98f0 2024-03-25T18:09:00.779 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.778+0000 7f64bcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:00.796 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.794+0000 7f64bcff9640 1 -- 172.21.15.66:0/3170178378 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 163638664 0 0) 0x7f64c00abde0 con 0x7f64c00a98f0 2024-03-25T18:09:00.796 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.794+0000 7f64bcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:09:00.796 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.794+0000 7f64bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:00.796 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.794+0000 7f64bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:00.796 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.794+0000 7f64bcff9640 10 monclient(hunting): my global_id is 4455 2024-03-25T18:09:00.796 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.795+0000 7f64bcff9640 1 -- 172.21.15.66:0/3170178378 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64ac004eb0 con 0x7f64c00a98f0 2024-03-25T18:09:00.827 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.826+0000 7f64bcff9640 1 -- 172.21.15.66:0/3170178378 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4282778022 0 0) 0x7f64ac004eb0 con 0x7f64c00a98f0 2024-03-25T18:09:00.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.826+0000 7f64bcff9640 1 -- 172.21.15.66:0/3170178378 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64c00abde0 con 0x7f64c00a98f0 2024-03-25T18:09:00.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.826+0000 7f64bcff9640 1 -- 172.21.15.66:0/3170178378 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f64b0001870 con 0x7f64c00a98f0 2024-03-25T18:09:00.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.826+0000 7f64bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:00.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.826+0000 7f64bcff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:09:00.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.826+0000 7f64bcff9640 10 monclient(hunting): dump: 2024-03-25T18:09:00.828 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:00.828 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:00.828 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:00.828 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:00.828 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:00.828 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:00.828 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:00.829 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:00.829 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:00.829 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:00.829 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.826+0000 7f64bcff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:00.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.842+0000 7f64bcff9640 1 -- 172.21.15.66:0/3170178378 <== mon.0 v1:172.21.15.66:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2687941107 0 0) 0x7f64c00abde0 con 0x7f64c00a98f0 2024-03-25T18:09:00.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.842+0000 7f64bcff9640 1 -- 172.21.15.66:0/3170178378 >> v1:172.21.15.66:6790/0 conn(0x7f64c013f710 legacy=0x7f64c01d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:00.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.842+0000 7f64bcff9640 1 -- 172.21.15.66:0/3170178378 >> v1:172.21.15.193:6789/0 conn(0x7f64c01430c0 legacy=0x7f64c01e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:00.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.842+0000 7f64bcff9640 10 monclient: _finish_hunting 0 2024-03-25T18:09:00.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.842+0000 7f64bcff9640 1 monclient: found mon.a 2024-03-25T18:09:00.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.842+0000 7f64bcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:00.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.842+0000 7f64bcff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:09:00.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.842+0000 7f64bcff9640 1 -- 172.21.15.66:0/3170178378 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c01d77a0 con 0x7f64c00a98f0 2024-03-25T18:09:00.846 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.842+0000 7f64bcff9640 10 monclient: _finish_auth 0 2024-03-25T18:09:00.846 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.842+0000 7f64bcff9640 10 monclient: _check_auth_tickets 2024-03-25T18:09:00.846 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.842+0000 7f64bcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:00.846 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.842+0000 7f64c67e2640 5 monclient: authenticate success, global_id 4455 2024-03-25T18:09:00.846 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.842+0000 7f64c67e2640 10 monclient: _renew_subs 2024-03-25T18:09:00.846 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.842+0000 7f64c67e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:09:00.846 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.842+0000 7f64c67e2640 1 -- 172.21.15.66:0/3170178378 --> v1:172.21.15.66:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64c0137f40 con 0x7f64c00a98f0 2024-03-25T18:09:00.846 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.842+0000 7f64c67e2640 10 client.4455.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:00.846 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.842+0000 7f64c67e2640 10 monclient: _renew_subs 2024-03-25T18:09:00.846 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.842+0000 7f64c67e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:09:00.846 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.842+0000 7f64c67e2640 1 -- 172.21.15.66:0/3170178378 --> v1:172.21.15.66:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64c01e34e0 con 0x7f64c00a98f0 2024-03-25T18:09:00.847 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.842+0000 7f64c67e2640 1 librados: init done 2024-03-25T18:09:00.847 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.843+0000 7f64c67e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:00.847 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.843+0000 7f64c67e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:00.847 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.843+0000 7f64c67e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:09:00.847 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:00.843+0000 7f64c67e2640 1 -- 172.21.15.66:0/3170178378 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64c01d75f0 con 0x7f64c00a98f0 2024-03-25T18:09:01.125 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.123+0000 7f64bcff9640 1 -- 172.21.15.66:0/3170178378 <== mon.0 v1:172.21.15.66:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64b00035f0 con 0x7f64c00a98f0 2024-03-25T18:09:01.125 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.123+0000 7f64bcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:01.125 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.123+0000 7f64bcff9640 1 -- 172.21.15.66:0/3170178378 <== mon.0 v1:172.21.15.66:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f64b0003900 con 0x7f64c00a98f0 2024-03-25T18:09:01.125 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.123+0000 7f64bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:01.125 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.123+0000 7f64bcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:01.125 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.123+0000 7f64bcff9640 10 monclient: dump: 2024-03-25T18:09:01.125 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:01.125 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:01.125 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:01.125 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:01.126 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:01.126 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:01.126 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:01.126 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:01.126 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:01.126 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:01.126 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.123+0000 7f64be7fc640 4 set_mon_vals no callback set 2024-03-25T18:09:01.127 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.124+0000 7f64bcff9640 1 -- 172.21.15.66:0/3170178378 <== mon.0 v1:172.21.15.66:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f64b00198d0 con 0x7f64c00a98f0 2024-03-25T18:09:01.127 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.125+0000 7f64bf7fe640 10 monclient: get_auth_request con 0x7f64ac06dcb0 auth_method 0 2024-03-25T18:09:01.127 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.125+0000 7f64bcff9640 10 client.4455.objecter ms_handle_connect 0x7f64ac06dcb0 2024-03-25T18:09:01.130 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.129+0000 7f64bcff9640 1 -- 172.21.15.66:0/3170178378 <== mon.0 v1:172.21.15.66:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f64b0053610 con 0x7f64c00a98f0 2024-03-25T18:09:01.130 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.129+0000 7f64bcff9640 10 client.4455.objecter ms_dispatch 0x7f64c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:01.131 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.129+0000 7f64bcff9640 3 client.4455.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:01.131 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.129+0000 7f64bcff9640 3 client.4455.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:01.131 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.129+0000 7f64bcff9640 20 client.4455.objecter dump_active .. 0 homeless 2024-03-25T18:09:01.131 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.129+0000 7f64bcff9640 1 -- 172.21.15.66:0/3170178378 <== mon.0 v1:172.21.15.66:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f64b004ef50 con 0x7f64c00a98f0 2024-03-25T18:09:01.131 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.129+0000 7f64bcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:01.131 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.129+0000 7f64bcff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:01.367 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.365+0000 7f64c67e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:01.367 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.365+0000 7f64c67e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:01.367 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.365+0000 7f64c67e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:09:01.367 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.365+0000 7f64c67e2640 1 -- 172.21.15.66:0/3170178378 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64c01d7930 con 0x7f64c00a98f0 2024-03-25T18:09:01.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.402+0000 7f64bcff9640 1 -- 172.21.15.66:0/3170178378 <== mon.0 v1:172.21.15.66:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f64b0051050 con 0x7f64c00a98f0 2024-03-25T18:09:01.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.402+0000 7f64bcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:01.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.402+0000 7f64bcff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:01.404 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:09:01.404 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:01.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.404+0000 7f64c67e2640 10 librados: watch_flush enter 2024-03-25T18:09:01.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.404+0000 7f64c67e2640 10 librados: watch_flush exit 2024-03-25T18:09:01.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.404+0000 7f64c67e2640 20 client.4455.objecter shutdown clearing up homeless session... 2024-03-25T18:09:01.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.404+0000 7f64c67e2640 10 client.4455.objecter successfully canceled tick 2024-03-25T18:09:01.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.404+0000 7f64c67e2640 1 -- 172.21.15.66:0/3170178378 >> v1:172.21.15.193:6800/2275676803 conn(0x7f64ac06dcb0 legacy=0x7f64ac070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:01.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.404+0000 7f64c67e2640 10 monclient: shutdown 2024-03-25T18:09:01.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.404+0000 7f64c67e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:01.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.404+0000 7f64c67e2640 1 -- 172.21.15.66:0/3170178378 >> v1:172.21.15.66:6789/0 conn(0x7f64c00a98f0 legacy=0x7f64c0138830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:01.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.405+0000 7f64c67e2640 1 -- 172.21.15.66:0/3170178378 shutdown_connections 2024-03-25T18:09:01.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.405+0000 7f64c67e2640 1 -- 172.21.15.66:0/3170178378 >> 172.21.15.66:0/3170178378 conn(0x7f64c00b7ea0 msgr2=0x7f64c0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:01.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.405+0000 7f64c67e2640 1 -- 172.21.15.66:0/3170178378 shutdown_connections 2024-03-25T18:09:01.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.405+0000 7f64c67e2640 1 -- 172.21.15.66:0/3170178378 wait complete. 2024-03-25T18:09:01.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:01.405+0000 7f64c67e2640 1 librados: shutdown 2024-03-25T18:09:01.417 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:08.223 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:08.347 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.345+0000 7f410aa0f640 10 monclient: get_monmap_and_config 2024-03-25T18:09:08.347 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.345+0000 7f410aa0f640 10 monclient: build_initial_monmap 2024-03-25T18:09:08.347 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.345+0000 7f410aa0f640 10 monclient: monmap: 2024-03-25T18:09:08.347 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:09:08.347 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:08.347 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:09:08.347107+0000 2024-03-25T18:09:08.347 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:09:08.347107+0000 2024-03-25T18:09:08.347 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:08.347 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:08.348 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:09:08.348 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:09:08.348 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:09:08.348 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:08.349 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.346+0000 7f410aa0f640 1 Processor -- start 2024-03-25T18:09:08.349 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.346+0000 7f410aa0f640 1 -- start start 2024-03-25T18:09:08.349 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.346+0000 7f410aa0f640 10 monclient: init 2024-03-25T18:09:08.349 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.347+0000 7f410aa0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:08.349 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.347+0000 7f410aa0f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:09:08.349 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.347+0000 7f410aa0f640 10 monclient(hunting): picked mon.noname-b con 0x7f41040a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:09:08.349 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.347+0000 7f410aa0f640 10 monclient(hunting): picked mon.noname-a con 0x7f410413f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:09:08.349 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.347+0000 7f410aa0f640 10 monclient(hunting): picked mon.noname-c con 0x7f41041430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:09:08.349 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.347+0000 7f410aa0f640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4104142520 con 0x7f410413f710 2024-03-25T18:09:08.349 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.347+0000 7f410aa0f640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41040b28a0 con 0x7f41041430c0 2024-03-25T18:09:08.349 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.347+0000 7f410aa0f640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41040b2a30 con 0x7f41040a98f0 2024-03-25T18:09:08.350 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.347+0000 7f410aa0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:08.350 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.347+0000 7f410aa0f640 10 monclient(hunting): authenticate will time out at 7297.407799s 2024-03-25T18:09:08.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.351+0000 7f4108fac640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f41041430c0 0x7f410414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:57516/0 (socket says 172.21.15.66:57516) 2024-03-25T18:09:08.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.351+0000 7f4108fac640 1 -- 172.21.15.66:0/840926204 learned_addr learned my addr 172.21.15.66:0/840926204 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:09:08.362 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.361+0000 7f4102ffd640 1 -- 172.21.15.66:0/840926204 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 625529820 0 0) 0x7f41040b28a0 con 0x7f41041430c0 2024-03-25T18:09:08.363 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.361+0000 7f4102ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:08.363 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.361+0000 7f4102ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:08.363 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.361+0000 7f4102ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:08.363 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.361+0000 7f4102ffd640 10 monclient(hunting): my global_id is 4450 2024-03-25T18:09:08.363 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.361+0000 7f4102ffd640 1 -- 172.21.15.66:0/840926204 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40ec003520 con 0x7f41041430c0 2024-03-25T18:09:08.380 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.378+0000 7f4102ffd640 1 -- 172.21.15.66:0/840926204 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4108565532 0 0) 0x7f40ec003520 con 0x7f41041430c0 2024-03-25T18:09:08.380 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.378+0000 7f4102ffd640 1 -- 172.21.15.66:0/840926204 >> v1:172.21.15.66:6790/0 conn(0x7f41040a98f0 legacy=0x7f41040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:08.380 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.378+0000 7f4102ffd640 1 -- 172.21.15.66:0/840926204 >> v1:172.21.15.66:6789/0 conn(0x7f410413f710 legacy=0x7f41041429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:08.380 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.378+0000 7f4102ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:09:08.381 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.378+0000 7f4102ffd640 1 monclient: found mon.noname-c 2024-03-25T18:09:08.381 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.378+0000 7f4102ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:08.381 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.379+0000 7f4102ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:09:08.381 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.379+0000 7f4102ffd640 1 -- 172.21.15.66:0/840926204 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41040b44c0 con 0x7f41041430c0 2024-03-25T18:09:08.381 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.379+0000 7f4102ffd640 10 monclient: _finish_auth 0 2024-03-25T18:09:08.381 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.379+0000 7f4102ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:09:08.381 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.379+0000 7f4102ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:08.381 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.379+0000 7f4102ffd640 1 -- 172.21.15.66:0/840926204 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f40fc001a10 con 0x7f41041430c0 2024-03-25T18:09:08.381 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.379+0000 7f4102ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:08.381 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.379+0000 7f4102ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:08.381 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.379+0000 7f4102ffd640 10 monclient: dump: 2024-03-25T18:09:08.381 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:08.381 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:08.381 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:08.381 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:08.381 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:08.381 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:08.382 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:08.382 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:08.382 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:08.382 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:08.382 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.379+0000 7f410aa0f640 5 monclient: authenticate success, global_id 4450 2024-03-25T18:09:08.382 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.379+0000 7f410aa0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:08.402 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.401+0000 7f4102ffd640 1 -- 172.21.15.66:0/840926204 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40fc001b70 con 0x7f41041430c0 2024-03-25T18:09:08.402 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.401+0000 7f4102ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:08.402 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.401+0000 7f4102ffd640 1 -- 172.21.15.66:0/840926204 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f40fc001f80 con 0x7f41041430c0 2024-03-25T18:09:08.402 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.401+0000 7f410aa0f640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:08.402 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.401+0000 7f410aa0f640 4 set_mon_vals no callback set 2024-03-25T18:09:08.402 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.401+0000 7f4102ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:08.402 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.401+0000 7f4102ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:08.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.401+0000 7f4102ffd640 10 monclient: dump: 2024-03-25T18:09:08.403 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:08.403 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:08.403 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:08.403 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:08.403 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:08.403 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:08.403 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:08.403 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:08.403 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:08.403 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:08.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.402+0000 7f410aa0f640 10 monclient: shutdown 2024-03-25T18:09:08.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.402+0000 7f410aa0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:08.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.402+0000 7f410aa0f640 1 -- 172.21.15.66:0/840926204 >> v1:172.21.15.193:6789/0 conn(0x7f41041430c0 legacy=0x7f410414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:08.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.402+0000 7f410aa0f640 1 -- 172.21.15.66:0/840926204 shutdown_connections 2024-03-25T18:09:08.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.402+0000 7f410aa0f640 1 -- 172.21.15.66:0/840926204 >> 172.21.15.66:0/840926204 conn(0x7f41040b7ea0 msgr2=0x7f41040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:08.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.402+0000 7f410aa0f640 1 -- 172.21.15.66:0/840926204 shutdown_connections 2024-03-25T18:09:08.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.402+0000 7f410aa0f640 1 -- 172.21.15.66:0/840926204 wait complete. 2024-03-25T18:09:08.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.403+0000 7f410aa0f640 10 monclient: build_initial_monmap 2024-03-25T18:09:08.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.403+0000 7f410aa0f640 10 monclient: monmap: 2024-03-25T18:09:08.405 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:09:08.405 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:08.405 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:09:08.406 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:09:08.406 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:08.406 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:09:08.406 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:09:08.406 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:09:08.406 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:09:08.406 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:08.406 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.403+0000 7f410aa0f640 1 librados: starting msgr at 2024-03-25T18:09:08.406 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.403+0000 7f410aa0f640 1 librados: starting objecter 2024-03-25T18:09:08.406 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.403+0000 7f410aa0f640 1 Processor -- start 2024-03-25T18:09:08.406 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.403+0000 7f410aa0f640 1 -- start start 2024-03-25T18:09:08.406 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.403+0000 7f410aa0f640 1 librados: setting wanted keys 2024-03-25T18:09:08.406 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.403+0000 7f410aa0f640 1 librados: calling monclient init 2024-03-25T18:09:08.406 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.403+0000 7f410aa0f640 10 monclient: init 2024-03-25T18:09:08.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.404+0000 7f410aa0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:08.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.404+0000 7f410aa0f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:09:08.408 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.404+0000 7f410aa0f640 10 monclient(hunting): picked mon.noname-b con 0x7f41040a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:09:08.408 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.404+0000 7f410aa0f640 10 monclient(hunting): picked mon.noname-c con 0x7f410413f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:09:08.408 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.404+0000 7f410aa0f640 10 monclient(hunting): picked mon.noname-a con 0x7f41041430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:09:08.408 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.404+0000 7f410aa0f640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41041db990 con 0x7f41041430c0 2024-03-25T18:09:08.408 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.404+0000 7f410aa0f640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410413afc0 con 0x7f41040a98f0 2024-03-25T18:09:08.408 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.404+0000 7f410aa0f640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410413b2a0 con 0x7f410413f710 2024-03-25T18:09:08.408 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.404+0000 7f410aa0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:08.408 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.404+0000 7f410aa0f640 10 monclient(hunting): authenticate will time out at 7297.464691s 2024-03-25T18:09:08.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.412+0000 7f4103fff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f41040a98f0 0x7f4104139d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:57582/0 (socket says 172.21.15.66:57582) 2024-03-25T18:09:08.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.412+0000 7f4103fff640 1 -- 172.21.15.66:0/3253906613 learned_addr learned my addr 172.21.15.66:0/3253906613 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:09:08.416 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.415+0000 7f4100ff9640 10 client.?.objecter ms_handle_connect 0x7f41040a98f0 2024-03-25T18:09:08.417 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.415+0000 7f4100ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:08.426 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.424+0000 7f4100ff9640 10 client.?.objecter ms_handle_connect 0x7f410413f710 2024-03-25T18:09:08.426 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.424+0000 7f4100ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:08.432 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.431+0000 7f4100ff9640 10 client.?.objecter ms_handle_connect 0x7f41041430c0 2024-03-25T18:09:08.432 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.431+0000 7f4100ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:08.434 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.433+0000 7f4100ff9640 1 -- 172.21.15.66:0/3253906613 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818485699 0 0) 0x7f410413b2a0 con 0x7f410413f710 2024-03-25T18:09:08.434 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.433+0000 7f4100ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:09:08.434 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.433+0000 7f4100ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:08.434 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.433+0000 7f4100ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:08.435 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.433+0000 7f4100ff9640 10 monclient(hunting): my global_id is 4454 2024-03-25T18:09:08.435 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.433+0000 7f4100ff9640 1 -- 172.21.15.66:0/3253906613 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40f0004eb0 con 0x7f410413f710 2024-03-25T18:09:08.439 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.438+0000 7f4100ff9640 1 -- 172.21.15.66:0/3253906613 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3250023260 0 0) 0x7f410413afc0 con 0x7f41040a98f0 2024-03-25T18:09:08.439 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.438+0000 7f4100ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:09:08.439 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.438+0000 7f4100ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:08.439 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.438+0000 7f4100ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:08.439 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.438+0000 7f4100ff9640 10 monclient(hunting): my global_id is 4453 2024-03-25T18:09:08.439 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.438+0000 7f4100ff9640 1 -- 172.21.15.66:0/3253906613 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f410413b2a0 con 0x7f41040a98f0 2024-03-25T18:09:08.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.442+0000 7f4100ff9640 1 -- 172.21.15.66:0/3253906613 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 521907428 0 0) 0x7f40f0004eb0 con 0x7f410413f710 2024-03-25T18:09:08.444 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.442+0000 7f4100ff9640 1 -- 172.21.15.66:0/3253906613 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f410413afc0 con 0x7f410413f710 2024-03-25T18:09:08.444 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.442+0000 7f4100ff9640 1 -- 172.21.15.66:0/3253906613 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f40f8001930 con 0x7f410413f710 2024-03-25T18:09:08.444 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.442+0000 7f4100ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:08.444 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.442+0000 7f4100ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:08.444 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.442+0000 7f4100ff9640 10 monclient(hunting): dump: 2024-03-25T18:09:08.444 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:08.444 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:08.444 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:08.444 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:08.444 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:08.444 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:08.444 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:08.444 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:08.444 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:08.445 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:08.445 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.442+0000 7f4100ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:08.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.452+0000 7f4100ff9640 1 -- 172.21.15.66:0/3253906613 <== mon.2 v1:172.21.15.66:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2659943811 0 0) 0x7f410413afc0 con 0x7f410413f710 2024-03-25T18:09:08.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.452+0000 7f4100ff9640 1 -- 172.21.15.66:0/3253906613 >> v1:172.21.15.193:6789/0 conn(0x7f41040a98f0 legacy=0x7f4104139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:08.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.452+0000 7f4100ff9640 1 -- 172.21.15.66:0/3253906613 >> v1:172.21.15.66:6789/0 conn(0x7f41041430c0 legacy=0x7f41041db280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:08.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.452+0000 7f4100ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:09:08.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.452+0000 7f4100ff9640 1 monclient: found mon.c 2024-03-25T18:09:08.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.452+0000 7f4100ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:08.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.452+0000 7f4100ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:09:08.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.452+0000 7f4100ff9640 1 -- 172.21.15.66:0/3253906613 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410413b580 con 0x7f410413f710 2024-03-25T18:09:08.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.452+0000 7f4100ff9640 10 monclient: _finish_auth 0 2024-03-25T18:09:08.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.452+0000 7f4100ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:09:08.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.452+0000 7f4100ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:08.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.452+0000 7f410aa0f640 5 monclient: authenticate success, global_id 4454 2024-03-25T18:09:08.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.452+0000 7f410aa0f640 10 monclient: _renew_subs 2024-03-25T18:09:08.455 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.452+0000 7f410aa0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:09:08.455 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.452+0000 7f410aa0f640 1 -- 172.21.15.66:0/3253906613 --> v1:172.21.15.66:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4104139930 con 0x7f410413f710 2024-03-25T18:09:08.455 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.452+0000 7f410aa0f640 10 client.4454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:08.455 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.452+0000 7f410aa0f640 10 monclient: _renew_subs 2024-03-25T18:09:08.455 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.452+0000 7f410aa0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:09:08.455 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.452+0000 7f410aa0f640 1 -- 172.21.15.66:0/3253906613 --> v1:172.21.15.66:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f410413b9a0 con 0x7f410413f710 2024-03-25T18:09:08.455 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.452+0000 7f410aa0f640 1 librados: init done 2024-03-25T18:09:08.455 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.453+0000 7f410aa0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:08.455 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.453+0000 7f410aa0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:08.456 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.453+0000 7f410aa0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:09:08.456 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.453+0000 7f410aa0f640 1 -- 172.21.15.66:0/3253906613 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41040627b0 con 0x7f410413f710 2024-03-25T18:09:08.770 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.769+0000 7f4100ff9640 1 -- 172.21.15.66:0/3253906613 <== mon.2 v1:172.21.15.66:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40f8003610 con 0x7f410413f710 2024-03-25T18:09:08.770 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.769+0000 7f4100ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:08.770 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.769+0000 7f4100ff9640 1 -- 172.21.15.66:0/3253906613 <== mon.2 v1:172.21.15.66:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f40f8003a20 con 0x7f410413f710 2024-03-25T18:09:08.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.769+0000 7f4100ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:08.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.769+0000 7f4100ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:08.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.769+0000 7f4100ff9640 10 monclient: dump: 2024-03-25T18:09:08.771 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:08.771 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:08.771 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:08.771 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:08.771 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:08.771 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:08.771 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:08.771 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:08.771 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:08.771 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:08.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.769+0000 7f41027fc640 4 set_mon_vals no callback set 2024-03-25T18:09:08.776 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.775+0000 7f4100ff9640 1 -- 172.21.15.66:0/3253906613 <== mon.2 v1:172.21.15.66:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f40f8003b80 con 0x7f410413f710 2024-03-25T18:09:08.777 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.776+0000 7f4103fff640 10 monclient: get_auth_request con 0x7f40f006dd10 auth_method 0 2024-03-25T18:09:08.778 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.776+0000 7f4100ff9640 10 client.4454.objecter ms_handle_connect 0x7f40f006dd10 2024-03-25T18:09:08.783 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.781+0000 7f4100ff9640 1 -- 172.21.15.66:0/3253906613 <== mon.2 v1:172.21.15.66:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f40f8053690 con 0x7f410413f710 2024-03-25T18:09:08.783 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.782+0000 7f4100ff9640 10 client.4454.objecter ms_dispatch 0x7f4104000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:08.783 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.782+0000 7f4100ff9640 3 client.4454.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:08.783 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.782+0000 7f4100ff9640 3 client.4454.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:08.783 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.782+0000 7f4100ff9640 20 client.4454.objecter dump_active .. 0 homeless 2024-03-25T18:09:08.783 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.782+0000 7f4100ff9640 1 -- 172.21.15.66:0/3253906613 <== mon.2 v1:172.21.15.66:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f40f804f1a0 con 0x7f410413f710 2024-03-25T18:09:08.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.782+0000 7f4100ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:08.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:08.782+0000 7f4100ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:09.009 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:09.008+0000 7f410aa0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:09.009 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:09.008+0000 7f410aa0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:09.009 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:09.008+0000 7f410aa0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:09:09.009 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:09.008+0000 7f410aa0f640 1 -- 172.21.15.66:0/3253906613 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4104069f40 con 0x7f410413f710 2024-03-25T18:09:09.084 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:09.083+0000 7f4100ff9640 1 -- 172.21.15.66:0/3253906613 <== mon.2 v1:172.21.15.66:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f40f80510d0 con 0x7f410413f710 2024-03-25T18:09:09.085 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:09.083+0000 7f4100ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:09.085 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:09.083+0000 7f4100ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:09.085 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:09:09.085 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:09.088 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:09.086+0000 7f410aa0f640 10 librados: watch_flush enter 2024-03-25T18:09:09.088 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:09.086+0000 7f410aa0f640 10 librados: watch_flush exit 2024-03-25T18:09:09.088 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:09.086+0000 7f410aa0f640 20 client.4454.objecter shutdown clearing up homeless session... 2024-03-25T18:09:09.088 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:09.086+0000 7f410aa0f640 10 client.4454.objecter successfully canceled tick 2024-03-25T18:09:09.088 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:09.086+0000 7f410aa0f640 1 -- 172.21.15.66:0/3253906613 >> v1:172.21.15.193:6800/2275676803 conn(0x7f40f006dd10 legacy=0x7f40f00701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:09.088 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:09.086+0000 7f410aa0f640 10 monclient: shutdown 2024-03-25T18:09:09.088 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:09.086+0000 7f410aa0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:09.088 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:09.086+0000 7f410aa0f640 1 -- 172.21.15.66:0/3253906613 >> v1:172.21.15.66:6790/0 conn(0x7f410413f710 legacy=0x7f41041dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:09.088 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:09.086+0000 7f410aa0f640 1 -- 172.21.15.66:0/3253906613 shutdown_connections 2024-03-25T18:09:09.088 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:09.086+0000 7f410aa0f640 1 -- 172.21.15.66:0/3253906613 >> 172.21.15.66:0/3253906613 conn(0x7f41040b7ea0 msgr2=0x7f410413e730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:09.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:09.086+0000 7f410aa0f640 1 -- 172.21.15.66:0/3253906613 shutdown_connections 2024-03-25T18:09:09.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:09.087+0000 7f410aa0f640 1 -- 172.21.15.66:0/3253906613 wait complete. 2024-03-25T18:09:09.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:09.087+0000 7f410aa0f640 1 librados: shutdown 2024-03-25T18:09:09.099 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:09.491 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:05:40.471 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:09.496 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:05:40.477 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:09.542 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:05:40.522 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:09.553 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:05:40.533 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:09.559 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:05:40.518 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:09.565 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:05:40.523 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:09.565 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:05:40.546 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:09.594 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:05:40.574 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:09.601 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:05:40.559 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:15.906 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:16.025 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.023+0000 7fcd91b63640 10 monclient: get_monmap_and_config 2024-03-25T18:09:16.025 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.023+0000 7fcd91b63640 10 monclient: build_initial_monmap 2024-03-25T18:09:16.025 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.023+0000 7fcd91b63640 10 monclient: monmap: 2024-03-25T18:09:16.025 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:09:16.025 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:16.025 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:09:16.025353+0000 2024-03-25T18:09:16.025 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:09:16.025353+0000 2024-03-25T18:09:16.025 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:16.026 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:16.026 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:09:16.026 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:09:16.026 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:09:16.026 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:16.026 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.025+0000 7fcd91b63640 1 Processor -- start 2024-03-25T18:09:16.027 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.025+0000 7fcd91b63640 1 -- start start 2024-03-25T18:09:16.027 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.025+0000 7fcd91b63640 10 monclient: init 2024-03-25T18:09:16.027 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.025+0000 7fcd91b63640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:16.027 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.025+0000 7fcd91b63640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:09:16.027 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.025+0000 7fcd91b63640 10 monclient(hunting): picked mon.noname-c con 0x7fcd8c0a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:09:16.027 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.025+0000 7fcd91b63640 10 monclient(hunting): picked mon.noname-a con 0x7fcd8c13f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:09:16.027 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.026+0000 7fcd91b63640 10 monclient(hunting): picked mon.noname-b con 0x7fcd8c1430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:09:16.027 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.026+0000 7fcd91b63640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8c142520 con 0x7fcd8c13f710 2024-03-25T18:09:16.028 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.026+0000 7fcd91b63640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8c0b28a0 con 0x7fcd8c0a98f0 2024-03-25T18:09:16.028 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.026+0000 7fcd91b63640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8c0b2a30 con 0x7fcd8c1430c0 2024-03-25T18:09:16.028 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.026+0000 7fcd91b63640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:16.028 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.026+0000 7fcd91b63640 10 monclient(hunting): authenticate will time out at 7305.086262s 2024-03-25T18:09:16.037 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.036+0000 7fcd8bfff640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7fcd8c1430c0 0x7fcd8c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:33440/0 (socket says 172.21.15.66:33440) 2024-03-25T18:09:16.038 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.036+0000 7fcd8bfff640 1 -- 172.21.15.66:0/310740786 learned_addr learned my addr 172.21.15.66:0/310740786 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:09:16.053 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.051+0000 7fcd8affd640 1 -- 172.21.15.66:0/310740786 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1195798417 0 0) 0x7fcd8c0b2a30 con 0x7fcd8c1430c0 2024-03-25T18:09:16.053 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.052+0000 7fcd8affd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:16.053 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.052+0000 7fcd8affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:16.053 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.052+0000 7fcd8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:16.053 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.052+0000 7fcd8affd640 10 monclient(hunting): my global_id is 4457 2024-03-25T18:09:16.053 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.052+0000 7fcd8affd640 1 -- 172.21.15.66:0/310740786 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd78003520 con 0x7fcd8c1430c0 2024-03-25T18:09:16.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.062+0000 7fcd8affd640 1 -- 172.21.15.66:0/310740786 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3068909929 0 0) 0x7fcd8c0b28a0 con 0x7fcd8c0a98f0 2024-03-25T18:09:16.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.062+0000 7fcd8affd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:16.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.062+0000 7fcd8affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:16.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.062+0000 7fcd8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:16.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.062+0000 7fcd8affd640 10 monclient(hunting): my global_id is 4456 2024-03-25T18:09:16.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.063+0000 7fcd8affd640 1 -- 172.21.15.66:0/310740786 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd8c0b2a30 con 0x7fcd8c0a98f0 2024-03-25T18:09:16.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.070+0000 7fcd8affd640 1 -- 172.21.15.66:0/310740786 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3696971953 0 0) 0x7fcd78003520 con 0x7fcd8c1430c0 2024-03-25T18:09:16.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.070+0000 7fcd8affd640 1 -- 172.21.15.66:0/310740786 >> v1:172.21.15.193:6789/0 conn(0x7fcd8c0a98f0 legacy=0x7fcd8c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:16.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.070+0000 7fcd8affd640 1 -- 172.21.15.66:0/310740786 >> v1:172.21.15.66:6789/0 conn(0x7fcd8c13f710 legacy=0x7fcd8c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:16.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.070+0000 7fcd8affd640 10 monclient: _finish_hunting 0 2024-03-25T18:09:16.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.070+0000 7fcd8affd640 1 monclient: found mon.noname-b 2024-03-25T18:09:16.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.070+0000 7fcd8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:16.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.070+0000 7fcd8affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.66:6790/0 2024-03-25T18:09:16.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.070+0000 7fcd8affd640 1 -- 172.21.15.66:0/310740786 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd8c0b44c0 con 0x7fcd8c1430c0 2024-03-25T18:09:16.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.070+0000 7fcd8affd640 10 monclient: _finish_auth 0 2024-03-25T18:09:16.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.070+0000 7fcd8affd640 10 monclient: _check_auth_tickets 2024-03-25T18:09:16.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.070+0000 7fcd8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:16.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.070+0000 7fcd8affd640 1 -- 172.21.15.66:0/310740786 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fcd84001a10 con 0x7fcd8c1430c0 2024-03-25T18:09:16.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.070+0000 7fcd91b63640 5 monclient: authenticate success, global_id 4457 2024-03-25T18:09:16.073 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.070+0000 7fcd91b63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:16.073 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.070+0000 7fcd8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:16.073 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.070+0000 7fcd8affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:16.073 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.070+0000 7fcd8affd640 10 monclient: dump: 2024-03-25T18:09:16.073 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:16.073 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:16.073 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:16.073 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:16.073 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:16.073 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:16.073 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:16.073 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:16.073 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:16.073 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:16.074 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.070+0000 7fcd91b63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:16.075 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.074+0000 7fcd8affd640 1 -- 172.21.15.66:0/310740786 <== mon.2 v1:172.21.15.66:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd84001b70 con 0x7fcd8c1430c0 2024-03-25T18:09:16.076 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.074+0000 7fcd8affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:16.076 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.074+0000 7fcd8affd640 1 -- 172.21.15.66:0/310740786 <== mon.2 v1:172.21.15.66:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fcd84001f80 con 0x7fcd8c1430c0 2024-03-25T18:09:16.076 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.074+0000 7fcd91b63640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:16.076 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.074+0000 7fcd91b63640 4 set_mon_vals no callback set 2024-03-25T18:09:16.076 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.074+0000 7fcd8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:16.076 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.074+0000 7fcd8affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:16.076 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.074+0000 7fcd8affd640 10 monclient: dump: 2024-03-25T18:09:16.076 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:16.076 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:16.076 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:16.076 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:16.076 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:16.076 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:16.077 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:16.077 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:16.077 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:16.077 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:16.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.075+0000 7fcd91b63640 10 monclient: shutdown 2024-03-25T18:09:16.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.075+0000 7fcd91b63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:16.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.075+0000 7fcd91b63640 1 -- 172.21.15.66:0/310740786 >> v1:172.21.15.66:6790/0 conn(0x7fcd8c1430c0 legacy=0x7fcd8c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:16.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.075+0000 7fcd91b63640 1 -- 172.21.15.66:0/310740786 shutdown_connections 2024-03-25T18:09:16.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.075+0000 7fcd91b63640 1 -- 172.21.15.66:0/310740786 >> 172.21.15.66:0/310740786 conn(0x7fcd8c0b7ea0 msgr2=0x7fcd8c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:16.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.075+0000 7fcd91b63640 1 -- 172.21.15.66:0/310740786 shutdown_connections 2024-03-25T18:09:16.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.075+0000 7fcd91b63640 1 -- 172.21.15.66:0/310740786 wait complete. 2024-03-25T18:09:16.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.076+0000 7fcd91b63640 10 monclient: build_initial_monmap 2024-03-25T18:09:16.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.076+0000 7fcd91b63640 10 monclient: monmap: 2024-03-25T18:09:16.079 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:09:16.079 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:16.079 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:09:16.079 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:09:16.079 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:16.079 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:09:16.079 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:09:16.079 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:09:16.079 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:09:16.079 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:16.079 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.076+0000 7fcd91b63640 1 librados: starting msgr at 2024-03-25T18:09:16.079 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.076+0000 7fcd91b63640 1 librados: starting objecter 2024-03-25T18:09:16.079 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.076+0000 7fcd91b63640 1 Processor -- start 2024-03-25T18:09:16.079 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.076+0000 7fcd91b63640 1 -- start start 2024-03-25T18:09:16.079 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.076+0000 7fcd91b63640 1 librados: setting wanted keys 2024-03-25T18:09:16.080 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.076+0000 7fcd91b63640 1 librados: calling monclient init 2024-03-25T18:09:16.080 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.076+0000 7fcd91b63640 10 monclient: init 2024-03-25T18:09:16.081 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.077+0000 7fcd91b63640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:16.081 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.077+0000 7fcd91b63640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:09:16.081 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.077+0000 7fcd91b63640 10 monclient(hunting): picked mon.noname-a con 0x7fcd8c0a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:09:16.081 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.077+0000 7fcd91b63640 10 monclient(hunting): picked mon.noname-c con 0x7fcd8c13f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:09:16.081 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.077+0000 7fcd91b63640 10 monclient(hunting): picked mon.noname-b con 0x7fcd8c1430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:09:16.081 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.077+0000 7fcd91b63640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8c1d8340 con 0x7fcd8c0a98f0 2024-03-25T18:09:16.081 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.077+0000 7fcd91b63640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8c1d7710 con 0x7fcd8c1430c0 2024-03-25T18:09:16.081 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.077+0000 7fcd91b63640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8c138e90 con 0x7fcd8c13f710 2024-03-25T18:09:16.081 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.077+0000 7fcd91b63640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:16.081 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.077+0000 7fcd91b63640 10 monclient(hunting): authenticate will time out at 7305.137753s 2024-03-25T18:09:16.083 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.082+0000 7fcd8bfff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fcd8c1430c0 0x7fcd8c133f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:44644/0 (socket says 172.21.15.66:44644) 2024-03-25T18:09:16.083 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.082+0000 7fcd8bfff640 1 -- 172.21.15.66:0/2605879559 learned_addr learned my addr 172.21.15.66:0/2605879559 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:09:16.084 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.083+0000 7fcd88ff9640 10 client.?.objecter ms_handle_connect 0x7fcd8c1430c0 2024-03-25T18:09:16.084 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.083+0000 7fcd88ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:16.101 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.100+0000 7fcd88ff9640 1 -- 172.21.15.66:0/2605879559 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 241662152 0 0) 0x7fcd8c1d7710 con 0x7fcd8c1430c0 2024-03-25T18:09:16.101 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.100+0000 7fcd88ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:09:16.101 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.100+0000 7fcd88ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:16.101 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.100+0000 7fcd88ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:16.101 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.100+0000 7fcd88ff9640 10 monclient(hunting): my global_id is 4459 2024-03-25T18:09:16.101 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.100+0000 7fcd88ff9640 1 -- 172.21.15.66:0/2605879559 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd6c004eb0 con 0x7fcd8c1430c0 2024-03-25T18:09:16.108 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.106+0000 7fcd88ff9640 10 client.?.objecter ms_handle_connect 0x7fcd8c0a98f0 2024-03-25T18:09:16.108 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.106+0000 7fcd88ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:16.114 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.112+0000 7fcd88ff9640 1 -- 172.21.15.66:0/2605879559 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1292960925 0 0) 0x7fcd6c004eb0 con 0x7fcd8c1430c0 2024-03-25T18:09:16.114 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.112+0000 7fcd88ff9640 1 -- 172.21.15.66:0/2605879559 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd8c1d7710 con 0x7fcd8c1430c0 2024-03-25T18:09:16.114 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.112+0000 7fcd88ff9640 1 -- 172.21.15.66:0/2605879559 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fcd84001ec0 con 0x7fcd8c1430c0 2024-03-25T18:09:16.114 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.112+0000 7fcd88ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:16.114 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.112+0000 7fcd88ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:16.114 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.112+0000 7fcd88ff9640 10 monclient(hunting): dump: 2024-03-25T18:09:16.114 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:16.114 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:16.115 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:16.115 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:16.115 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:16.115 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:16.115 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:16.115 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:16.115 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:16.115 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:16.115 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.112+0000 7fcd88ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:16.116 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.114+0000 7fcd88ff9640 10 client.?.objecter ms_handle_connect 0x7fcd8c13f710 2024-03-25T18:09:16.116 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.114+0000 7fcd88ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:16.126 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.125+0000 7fcd88ff9640 1 -- 172.21.15.66:0/2605879559 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1667100004 0 0) 0x7fcd8c138e90 con 0x7fcd8c13f710 2024-03-25T18:09:16.127 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.125+0000 7fcd88ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:09:16.127 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.125+0000 7fcd88ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:16.127 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.125+0000 7fcd88ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:16.127 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.125+0000 7fcd88ff9640 10 monclient(hunting): my global_id is 4460 2024-03-25T18:09:16.127 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.125+0000 7fcd88ff9640 1 -- 172.21.15.66:0/2605879559 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd6c004eb0 con 0x7fcd8c13f710 2024-03-25T18:09:16.136 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.135+0000 7fcd88ff9640 1 -- 172.21.15.66:0/2605879559 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2176534202 0 0) 0x7fcd8c1d7710 con 0x7fcd8c1430c0 2024-03-25T18:09:16.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.135+0000 7fcd88ff9640 1 -- 172.21.15.66:0/2605879559 >> v1:172.21.15.66:6790/0 conn(0x7fcd8c13f710 legacy=0x7fcd8c138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:16.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.135+0000 7fcd88ff9640 1 -- 172.21.15.66:0/2605879559 >> v1:172.21.15.66:6789/0 conn(0x7fcd8c0a98f0 legacy=0x7fcd8c1d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:16.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.135+0000 7fcd88ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:09:16.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.135+0000 7fcd88ff9640 1 monclient: found mon.b 2024-03-25T18:09:16.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.135+0000 7fcd88ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:16.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.135+0000 7fcd88ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:09:16.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.135+0000 7fcd88ff9640 1 -- 172.21.15.66:0/2605879559 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd8c139020 con 0x7fcd8c1430c0 2024-03-25T18:09:16.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.135+0000 7fcd88ff9640 10 monclient: _finish_auth 0 2024-03-25T18:09:16.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.135+0000 7fcd88ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:09:16.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.135+0000 7fcd88ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:16.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.135+0000 7fcd91b63640 5 monclient: authenticate success, global_id 4459 2024-03-25T18:09:16.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.135+0000 7fcd91b63640 10 monclient: _renew_subs 2024-03-25T18:09:16.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.135+0000 7fcd91b63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:09:16.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.135+0000 7fcd91b63640 1 -- 172.21.15.66:0/2605879559 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd8c137f30 con 0x7fcd8c1430c0 2024-03-25T18:09:16.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.135+0000 7fcd91b63640 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:16.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.135+0000 7fcd91b63640 10 monclient: _renew_subs 2024-03-25T18:09:16.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.135+0000 7fcd91b63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:09:16.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.135+0000 7fcd91b63640 1 -- 172.21.15.66:0/2605879559 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd8c1352b0 con 0x7fcd8c1430c0 2024-03-25T18:09:16.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.135+0000 7fcd91b63640 1 librados: init done 2024-03-25T18:09:16.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.136+0000 7fcd91b63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:16.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.136+0000 7fcd91b63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:16.139 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.136+0000 7fcd91b63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:09:16.139 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.136+0000 7fcd91b63640 1 -- 172.21.15.66:0/2605879559 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd8c1d78a0 con 0x7fcd8c1430c0 2024-03-25T18:09:16.410 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.409+0000 7fcd88ff9640 1 -- 172.21.15.66:0/2605879559 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd84004e10 con 0x7fcd8c1430c0 2024-03-25T18:09:16.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.409+0000 7fcd88ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:16.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.409+0000 7fcd88ff9640 1 -- 172.21.15.66:0/2605879559 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fcd84002020 con 0x7fcd8c1430c0 2024-03-25T18:09:16.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.409+0000 7fcd88ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:16.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.409+0000 7fcd88ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:16.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.409+0000 7fcd88ff9640 10 monclient: dump: 2024-03-25T18:09:16.411 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:16.411 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:16.411 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:16.411 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:16.411 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:16.411 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:16.411 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:16.411 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:16.411 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:16.412 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:16.412 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.409+0000 7fcd8a7fc640 4 set_mon_vals no callback set 2024-03-25T18:09:16.412 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.410+0000 7fcd88ff9640 1 -- 172.21.15.66:0/2605879559 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7fcd8401af90 con 0x7fcd8c1430c0 2024-03-25T18:09:16.412 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.410+0000 7fcd8b7fe640 10 monclient: get_auth_request con 0x7fcd6c06dd40 auth_method 0 2024-03-25T18:09:16.412 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.411+0000 7fcd88ff9640 10 client.4459.objecter ms_handle_connect 0x7fcd6c06dd40 2024-03-25T18:09:16.417 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.415+0000 7fcd88ff9640 1 -- 172.21.15.66:0/2605879559 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7fcd84054ba0 con 0x7fcd8c1430c0 2024-03-25T18:09:16.417 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.415+0000 7fcd88ff9640 10 client.4459.objecter ms_dispatch 0x7fcd8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:16.417 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.415+0000 7fcd88ff9640 3 client.4459.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:16.417 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.415+0000 7fcd88ff9640 3 client.4459.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:16.417 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.415+0000 7fcd88ff9640 20 client.4459.objecter dump_active .. 0 homeless 2024-03-25T18:09:16.417 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.415+0000 7fcd88ff9640 1 -- 172.21.15.66:0/2605879559 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcd840504e0 con 0x7fcd8c1430c0 2024-03-25T18:09:16.417 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.415+0000 7fcd88ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:16.417 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.415+0000 7fcd88ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:16.669 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.667+0000 7fcd91b63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:16.669 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.667+0000 7fcd91b63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:16.669 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.667+0000 7fcd91b63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:09:16.669 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.667+0000 7fcd91b63640 1 -- 172.21.15.66:0/2605879559 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd8c1d7500 con 0x7fcd8c1430c0 2024-03-25T18:09:16.818 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.816+0000 7fcd88ff9640 1 -- 172.21.15.66:0/2605879559 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7fcd840525e0 con 0x7fcd8c1430c0 2024-03-25T18:09:16.818 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.816+0000 7fcd88ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:16.818 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.816+0000 7fcd88ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:16.818 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:09:16.819 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:16.821 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.818+0000 7fcd91b63640 10 librados: watch_flush enter 2024-03-25T18:09:16.821 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.818+0000 7fcd91b63640 10 librados: watch_flush exit 2024-03-25T18:09:16.821 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.818+0000 7fcd91b63640 20 client.4459.objecter shutdown clearing up homeless session... 2024-03-25T18:09:16.821 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.818+0000 7fcd91b63640 10 client.4459.objecter successfully canceled tick 2024-03-25T18:09:16.821 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.819+0000 7fcd91b63640 1 -- 172.21.15.66:0/2605879559 >> v1:172.21.15.193:6800/2275676803 conn(0x7fcd6c06dd40 legacy=0x7fcd6c070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:16.821 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.819+0000 7fcd91b63640 10 monclient: shutdown 2024-03-25T18:09:16.821 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.819+0000 7fcd91b63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:16.821 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.819+0000 7fcd91b63640 1 -- 172.21.15.66:0/2605879559 >> v1:172.21.15.193:6789/0 conn(0x7fcd8c1430c0 legacy=0x7fcd8c133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:16.821 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.819+0000 7fcd91b63640 1 -- 172.21.15.66:0/2605879559 shutdown_connections 2024-03-25T18:09:16.822 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.819+0000 7fcd91b63640 1 -- 172.21.15.66:0/2605879559 >> 172.21.15.66:0/2605879559 conn(0x7fcd8c0b7ea0 msgr2=0x7fcd8c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:16.822 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.819+0000 7fcd91b63640 1 -- 172.21.15.66:0/2605879559 shutdown_connections 2024-03-25T18:09:16.822 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.819+0000 7fcd91b63640 1 -- 172.21.15.66:0/2605879559 wait complete. 2024-03-25T18:09:16.822 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:16.819+0000 7fcd91b63640 1 librados: shutdown 2024-03-25T18:09:16.834 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:23.672 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:23.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.791+0000 7f53d5072640 10 monclient: get_monmap_and_config 2024-03-25T18:09:23.793 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.791+0000 7f53d5072640 10 monclient: build_initial_monmap 2024-03-25T18:09:23.793 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.791+0000 7f53d5072640 10 monclient: monmap: 2024-03-25T18:09:23.793 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:09:23.793 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:23.793 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:09:23.792597+0000 2024-03-25T18:09:23.793 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:09:23.792597+0000 2024-03-25T18:09:23.793 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:23.793 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:23.793 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:09:23.793 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:09:23.793 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:09:23.793 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:23.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.792+0000 7f53d5072640 1 Processor -- start 2024-03-25T18:09:23.795 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.792+0000 7f53d5072640 1 -- start start 2024-03-25T18:09:23.795 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.792+0000 7f53d5072640 10 monclient: init 2024-03-25T18:09:23.795 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.792+0000 7f53d5072640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:23.795 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.792+0000 7f53d5072640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:09:23.795 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.793+0000 7f53d5072640 10 monclient(hunting): picked mon.noname-c con 0x7f53d00a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:09:23.795 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.793+0000 7f53d5072640 10 monclient(hunting): picked mon.noname-b con 0x7f53d013f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:09:23.795 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.793+0000 7f53d5072640 10 monclient(hunting): picked mon.noname-a con 0x7f53d01430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:09:23.795 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.793+0000 7f53d5072640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53d0142520 con 0x7f53d01430c0 2024-03-25T18:09:23.795 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.793+0000 7f53d5072640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53d00b28a0 con 0x7f53d00a98f0 2024-03-25T18:09:23.795 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.793+0000 7f53d5072640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53d00b2a30 con 0x7f53d013f710 2024-03-25T18:09:23.795 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.793+0000 7f53d5072640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:23.795 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.793+0000 7f53d5072640 10 monclient(hunting): authenticate will time out at 7312.853191s 2024-03-25T18:09:23.808 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.807+0000 7f53ced9d640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f53d00a98f0 0x7f53d00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:47072/0 (socket says 172.21.15.66:47072) 2024-03-25T18:09:23.808 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.807+0000 7f53ced9d640 1 -- 172.21.15.66:0/1578610010 learned_addr learned my addr 172.21.15.66:0/1578610010 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:09:23.826 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.825+0000 7f53cdd9b640 1 -- 172.21.15.66:0/1578610010 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520961012 0 0) 0x7f53d00b28a0 con 0x7f53d00a98f0 2024-03-25T18:09:23.826 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.825+0000 7f53cdd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:09:23.826 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.825+0000 7f53cdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:23.826 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.825+0000 7f53cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:23.827 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.825+0000 7f53cdd9b640 10 monclient(hunting): my global_id is 4462 2024-03-25T18:09:23.827 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.825+0000 7f53cdd9b640 1 -- 172.21.15.66:0/1578610010 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53b8003520 con 0x7f53d00a98f0 2024-03-25T18:09:23.835 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.834+0000 7f53cdd9b640 1 -- 172.21.15.66:0/1578610010 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2396433468 0 0) 0x7f53d00b2a30 con 0x7f53d013f710 2024-03-25T18:09:23.836 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.834+0000 7f53cdd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:09:23.836 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.834+0000 7f53cdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:23.836 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.834+0000 7f53cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:23.836 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.834+0000 7f53cdd9b640 10 monclient(hunting): my global_id is 4463 2024-03-25T18:09:23.836 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.834+0000 7f53cdd9b640 1 -- 172.21.15.66:0/1578610010 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53d00b28a0 con 0x7f53d013f710 2024-03-25T18:09:23.836 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.835+0000 7f53cdd9b640 1 -- 172.21.15.66:0/1578610010 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1951489809 0 0) 0x7f53d0142520 con 0x7f53d01430c0 2024-03-25T18:09:23.836 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.835+0000 7f53cdd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:09:23.836 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.835+0000 7f53cdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:23.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.835+0000 7f53cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:23.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.835+0000 7f53cdd9b640 10 monclient(hunting): my global_id is 4470 2024-03-25T18:09:23.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.835+0000 7f53cdd9b640 1 -- 172.21.15.66:0/1578610010 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53d00b2a30 con 0x7f53d01430c0 2024-03-25T18:09:23.844 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.842+0000 7f53cdd9b640 1 -- 172.21.15.66:0/1578610010 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2816465681 0 0) 0x7f53b8003520 con 0x7f53d00a98f0 2024-03-25T18:09:23.844 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.842+0000 7f53cdd9b640 1 -- 172.21.15.66:0/1578610010 >> v1:172.21.15.66:6790/0 conn(0x7f53d013f710 legacy=0x7f53d01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:23.844 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.842+0000 7f53cdd9b640 1 -- 172.21.15.66:0/1578610010 >> v1:172.21.15.66:6789/0 conn(0x7f53d01430c0 legacy=0x7f53d014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:23.844 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.842+0000 7f53cdd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:09:23.844 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.842+0000 7f53cdd9b640 1 monclient: found mon.noname-c 2024-03-25T18:09:23.844 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.842+0000 7f53cdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:23.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.842+0000 7f53cdd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:09:23.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.842+0000 7f53cdd9b640 1 -- 172.21.15.66:0/1578610010 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d00b44c0 con 0x7f53d00a98f0 2024-03-25T18:09:23.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.842+0000 7f53cdd9b640 10 monclient: _finish_auth 0 2024-03-25T18:09:23.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.842+0000 7f53cdd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:09:23.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.842+0000 7f53cdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:23.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.842+0000 7f53cdd9b640 1 -- 172.21.15.66:0/1578610010 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f53c8001a10 con 0x7f53d00a98f0 2024-03-25T18:09:23.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.842+0000 7f53d5072640 5 monclient: authenticate success, global_id 4462 2024-03-25T18:09:23.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.842+0000 7f53d5072640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:23.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.842+0000 7f53cdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:23.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.842+0000 7f53cdd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:23.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.842+0000 7f53cdd9b640 10 monclient: dump: 2024-03-25T18:09:23.845 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:23.845 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:23.845 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:23.845 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:23.846 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:23.846 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:23.846 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:23.846 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:23.846 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:23.846 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:23.846 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.842+0000 7f53d5072640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:23.864 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.863+0000 7f53cdd9b640 1 -- 172.21.15.66:0/1578610010 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53c8001b70 con 0x7f53d00a98f0 2024-03-25T18:09:23.865 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.863+0000 7f53cdd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:23.865 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.863+0000 7f53cdd9b640 1 -- 172.21.15.66:0/1578610010 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f53c8001f80 con 0x7f53d00a98f0 2024-03-25T18:09:23.865 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.863+0000 7f53cdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:23.865 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.863+0000 7f53cdd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:23.865 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.863+0000 7f53cdd9b640 10 monclient: dump: 2024-03-25T18:09:23.865 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:23.865 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:23.865 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:23.865 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:23.865 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:23.865 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:23.865 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:23.865 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:23.865 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:23.866 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:23.866 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.863+0000 7f53d5072640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:23.866 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.863+0000 7f53d5072640 4 set_mon_vals no callback set 2024-03-25T18:09:23.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.864+0000 7f53d5072640 10 monclient: shutdown 2024-03-25T18:09:23.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.864+0000 7f53d5072640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:23.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.864+0000 7f53d5072640 1 -- 172.21.15.66:0/1578610010 >> v1:172.21.15.193:6789/0 conn(0x7f53d00a98f0 legacy=0x7f53d00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:23.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.864+0000 7f53d5072640 1 -- 172.21.15.66:0/1578610010 shutdown_connections 2024-03-25T18:09:23.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.864+0000 7f53d5072640 1 -- 172.21.15.66:0/1578610010 >> 172.21.15.66:0/1578610010 conn(0x7f53d00b7ea0 msgr2=0x7f53d00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:23.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.864+0000 7f53d5072640 1 -- 172.21.15.66:0/1578610010 shutdown_connections 2024-03-25T18:09:23.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.864+0000 7f53d5072640 1 -- 172.21.15.66:0/1578610010 wait complete. 2024-03-25T18:09:23.868 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.865+0000 7f53d5072640 10 monclient: build_initial_monmap 2024-03-25T18:09:23.868 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.865+0000 7f53d5072640 10 monclient: monmap: 2024-03-25T18:09:23.868 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:09:23.868 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:23.868 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:09:23.868 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:09:23.868 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:23.868 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:09:23.868 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:09:23.868 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:09:23.868 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:09:23.868 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:23.868 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.865+0000 7f53d5072640 1 librados: starting msgr at 2024-03-25T18:09:23.868 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.865+0000 7f53d5072640 1 librados: starting objecter 2024-03-25T18:09:23.868 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.866+0000 7f53d5072640 1 Processor -- start 2024-03-25T18:09:23.868 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.866+0000 7f53d5072640 1 -- start start 2024-03-25T18:09:23.869 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.866+0000 7f53d5072640 1 librados: setting wanted keys 2024-03-25T18:09:23.869 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.866+0000 7f53d5072640 1 librados: calling monclient init 2024-03-25T18:09:23.869 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.866+0000 7f53d5072640 10 monclient: init 2024-03-25T18:09:23.869 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.866+0000 7f53d5072640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:23.869 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.866+0000 7f53d5072640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:09:23.869 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.866+0000 7f53d5072640 10 monclient(hunting): picked mon.noname-b con 0x7f53d00a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:09:23.870 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.866+0000 7f53d5072640 10 monclient(hunting): picked mon.noname-c con 0x7f53d013f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:09:23.870 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.866+0000 7f53d5072640 10 monclient(hunting): picked mon.noname-a con 0x7f53d01430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:09:23.870 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.866+0000 7f53d5072640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53d014e550 con 0x7f53d01430c0 2024-03-25T18:09:23.870 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.866+0000 7f53d5072640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53d006eed0 con 0x7f53d00a98f0 2024-03-25T18:09:23.870 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.866+0000 7f53d5072640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53d006f1b0 con 0x7f53d013f710 2024-03-25T18:09:23.870 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.866+0000 7f53d5072640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:23.870 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.866+0000 7f53d5072640 10 monclient(hunting): authenticate will time out at 7312.926812s 2024-03-25T18:09:23.880 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.878+0000 7f53cf59e640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f53d01430c0 0x7f53d01e66f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:50686/0 (socket says 172.21.15.66:50686) 2024-03-25T18:09:23.880 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.878+0000 7f53cf59e640 1 -- 172.21.15.66:0/3827445951 learned_addr learned my addr 172.21.15.66:0/3827445951 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:09:23.881 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.880+0000 7f53b77fe640 10 client.?.objecter ms_handle_connect 0x7f53d00a98f0 2024-03-25T18:09:23.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.880+0000 7f53b77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:23.885 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.884+0000 7f53b77fe640 10 client.?.objecter ms_handle_connect 0x7f53d01430c0 2024-03-25T18:09:23.885 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.884+0000 7f53b77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:23.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.891+0000 7f53b77fe640 1 -- 172.21.15.66:0/3827445951 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 745763814 0 0) 0x7f53d014e550 con 0x7f53d01430c0 2024-03-25T18:09:23.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.891+0000 7f53b77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:23.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.891+0000 7f53b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:23.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.891+0000 7f53b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:23.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.891+0000 7f53b77fe640 10 monclient(hunting): my global_id is 4473 2024-03-25T18:09:23.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.891+0000 7f53b77fe640 1 -- 172.21.15.66:0/3827445951 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53bc004eb0 con 0x7f53d01430c0 2024-03-25T18:09:23.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.891+0000 7f53b77fe640 1 -- 172.21.15.66:0/3827445951 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268899838 0 0) 0x7f53d006eed0 con 0x7f53d00a98f0 2024-03-25T18:09:23.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.891+0000 7f53b77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:23.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.891+0000 7f53b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:23.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.891+0000 7f53b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:23.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.891+0000 7f53b77fe640 10 monclient(hunting): my global_id is 4465 2024-03-25T18:09:23.893 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.891+0000 7f53b77fe640 1 -- 172.21.15.66:0/3827445951 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53d014e550 con 0x7f53d00a98f0 2024-03-25T18:09:23.899 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.898+0000 7f53b77fe640 10 client.?.objecter ms_handle_connect 0x7f53d013f710 2024-03-25T18:09:23.900 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.898+0000 7f53b77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:23.902 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.901+0000 7f53b77fe640 1 -- 172.21.15.66:0/3827445951 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1180640272 0 0) 0x7f53bc004eb0 con 0x7f53d01430c0 2024-03-25T18:09:23.903 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.901+0000 7f53b77fe640 1 -- 172.21.15.66:0/3827445951 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53d006eed0 con 0x7f53d01430c0 2024-03-25T18:09:23.903 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.901+0000 7f53b77fe640 1 -- 172.21.15.66:0/3827445951 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f53c40018f0 con 0x7f53d01430c0 2024-03-25T18:09:23.903 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.901+0000 7f53b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:23.903 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.901+0000 7f53b77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:09:23.903 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.901+0000 7f53b77fe640 10 monclient(hunting): dump: 2024-03-25T18:09:23.903 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:23.903 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:23.903 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:23.903 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:23.903 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:23.903 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:23.903 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:23.903 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:23.903 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:23.903 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:23.904 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.901+0000 7f53b77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:23.915 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.914+0000 7f53b77fe640 1 -- 172.21.15.66:0/3827445951 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4273719331 0 0) 0x7f53d014e550 con 0x7f53d00a98f0 2024-03-25T18:09:23.915 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.914+0000 7f53b77fe640 1 -- 172.21.15.66:0/3827445951 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53bc004eb0 con 0x7f53d00a98f0 2024-03-25T18:09:23.915 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.914+0000 7f53b77fe640 1 -- 172.21.15.66:0/3827445951 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f53c8001e80 con 0x7f53d00a98f0 2024-03-25T18:09:23.915 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.914+0000 7f53b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:23.915 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.914+0000 7f53b77fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:23.916 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.914+0000 7f53b77fe640 10 monclient(hunting): dump: 2024-03-25T18:09:23.916 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:23.916 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:23.916 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:23.916 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:23.916 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:23.916 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:23.916 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:23.916 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:23.916 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:23.916 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:23.917 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.915+0000 7f53b77fe640 1 -- 172.21.15.66:0/3827445951 <== mon.0 v1:172.21.15.66:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1750875050 0 0) 0x7f53d006eed0 con 0x7f53d01430c0 2024-03-25T18:09:23.917 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.915+0000 7f53b77fe640 1 -- 172.21.15.66:0/3827445951 >> v1:172.21.15.66:6790/0 conn(0x7f53d013f710 legacy=0x7f53d006f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:23.917 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.915+0000 7f53b77fe640 1 -- 172.21.15.66:0/3827445951 >> v1:172.21.15.193:6789/0 conn(0x7f53d00a98f0 legacy=0x7f53d014dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:23.917 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.915+0000 7f53b77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:09:23.917 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.915+0000 7f53b77fe640 1 monclient: found mon.a 2024-03-25T18:09:23.917 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.915+0000 7f53b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:23.917 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.915+0000 7f53b77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:09:23.917 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.915+0000 7f53b77fe640 1 -- 172.21.15.66:0/3827445951 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d014e9e0 con 0x7f53d01430c0 2024-03-25T18:09:23.917 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.915+0000 7f53b77fe640 10 monclient: _finish_auth 0 2024-03-25T18:09:23.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.915+0000 7f53b77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:09:23.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.915+0000 7f53b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:23.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.915+0000 7f53d5072640 5 monclient: authenticate success, global_id 4473 2024-03-25T18:09:23.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.915+0000 7f53d5072640 10 monclient: _renew_subs 2024-03-25T18:09:23.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.915+0000 7f53d5072640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:09:23.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.915+0000 7f53d5072640 1 -- 172.21.15.66:0/3827445951 --> v1:172.21.15.66:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53d014eb70 con 0x7f53d01430c0 2024-03-25T18:09:23.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.915+0000 7f53d5072640 10 client.4473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:23.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.915+0000 7f53d5072640 10 monclient: _renew_subs 2024-03-25T18:09:23.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.915+0000 7f53d5072640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:09:23.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.915+0000 7f53d5072640 1 -- 172.21.15.66:0/3827445951 --> v1:172.21.15.66:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53d00701a0 con 0x7f53d01430c0 2024-03-25T18:09:23.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.915+0000 7f53d5072640 1 librados: init done 2024-03-25T18:09:23.919 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.916+0000 7f53d5072640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:23.919 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.916+0000 7f53d5072640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:23.919 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.916+0000 7f53d5072640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:09:23.919 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:23.916+0000 7f53d5072640 1 -- 172.21.15.66:0/3827445951 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53d00627b0 con 0x7f53d01430c0 2024-03-25T18:09:24.228 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.226+0000 7f53b77fe640 1 -- 172.21.15.66:0/3827445951 <== mon.0 v1:172.21.15.66:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53c4003610 con 0x7f53d01430c0 2024-03-25T18:09:24.228 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.226+0000 7f53b77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:24.228 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.226+0000 7f53b77fe640 1 -- 172.21.15.66:0/3827445951 <== mon.0 v1:172.21.15.66:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f53c4003a20 con 0x7f53d01430c0 2024-03-25T18:09:24.228 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.226+0000 7f53b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:24.228 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.226+0000 7f53b77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:24.228 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.226+0000 7f53b77fe640 10 monclient: dump: 2024-03-25T18:09:24.228 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:24.228 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:24.228 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:24.228 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:24.228 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:24.228 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:24.228 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:24.228 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:24.229 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:24.229 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:24.229 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.226+0000 7f53ccd99640 4 set_mon_vals no callback set 2024-03-25T18:09:24.229 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.227+0000 7f53b77fe640 1 -- 172.21.15.66:0/3827445951 <== mon.0 v1:172.21.15.66:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f53c4003c40 con 0x7f53d01430c0 2024-03-25T18:09:24.229 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.228+0000 7f53ced9d640 10 monclient: get_auth_request con 0x7f53bc06de90 auth_method 0 2024-03-25T18:09:24.229 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.228+0000 7f53b77fe640 10 client.4473.objecter ms_handle_connect 0x7f53bc06de90 2024-03-25T18:09:24.234 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.232+0000 7f53b77fe640 1 -- 172.21.15.66:0/3827445951 <== mon.0 v1:172.21.15.66:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f53c4054750 con 0x7f53d01430c0 2024-03-25T18:09:24.234 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.232+0000 7f53b77fe640 10 client.4473.objecter ms_dispatch 0x7f53d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:24.234 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.232+0000 7f53b77fe640 3 client.4473.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:24.234 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.232+0000 7f53b77fe640 3 client.4473.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:24.234 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.233+0000 7f53b77fe640 20 client.4473.objecter dump_active .. 0 homeless 2024-03-25T18:09:24.234 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.233+0000 7f53b77fe640 1 -- 172.21.15.66:0/3827445951 <== mon.0 v1:172.21.15.66:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f53c4050260 con 0x7f53d01430c0 2024-03-25T18:09:24.235 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.233+0000 7f53b77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:24.235 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.233+0000 7f53b77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:24.488 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.487+0000 7f53d5072640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:24.489 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.487+0000 7f53d5072640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:24.489 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.487+0000 7f53d5072640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:09:24.489 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.487+0000 7f53d5072640 1 -- 172.21.15.66:0/3827445951 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53d013df50 con 0x7f53d01430c0 2024-03-25T18:09:24.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.515+0000 7f53b77fe640 1 -- 172.21.15.66:0/3827445951 <== mon.0 v1:172.21.15.66:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f53c4052190 con 0x7f53d01430c0 2024-03-25T18:09:24.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.515+0000 7f53b77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:24.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.515+0000 7f53b77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:24.517 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:09:24.517 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:24.520 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.518+0000 7f53d5072640 10 librados: watch_flush enter 2024-03-25T18:09:24.520 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.518+0000 7f53d5072640 10 librados: watch_flush exit 2024-03-25T18:09:24.520 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.518+0000 7f53d5072640 20 client.4473.objecter shutdown clearing up homeless session... 2024-03-25T18:09:24.520 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.518+0000 7f53d5072640 10 client.4473.objecter successfully canceled tick 2024-03-25T18:09:24.520 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.518+0000 7f53d5072640 1 -- 172.21.15.66:0/3827445951 >> v1:172.21.15.193:6800/2275676803 conn(0x7f53bc06de90 legacy=0x7f53bc070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:24.520 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.518+0000 7f53d5072640 10 monclient: shutdown 2024-03-25T18:09:24.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.518+0000 7f53d5072640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:24.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.518+0000 7f53d5072640 1 -- 172.21.15.66:0/3827445951 >> v1:172.21.15.66:6789/0 conn(0x7f53d01430c0 legacy=0x7f53d01e66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:24.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.518+0000 7f53d5072640 1 -- 172.21.15.66:0/3827445951 shutdown_connections 2024-03-25T18:09:24.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.518+0000 7f53d5072640 1 -- 172.21.15.66:0/3827445951 >> 172.21.15.66:0/3827445951 conn(0x7f53d00b7ea0 msgr2=0x7f53d0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:24.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.518+0000 7f53d5072640 1 -- 172.21.15.66:0/3827445951 shutdown_connections 2024-03-25T18:09:24.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.518+0000 7f53d5072640 1 -- 172.21.15.66:0/3827445951 wait complete. 2024-03-25T18:09:24.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:24.519+0000 7f53d5072640 1 librados: shutdown 2024-03-25T18:09:24.532 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:29.663 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:06:00.643 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:29.724 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:06:00.704 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:29.750 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:06:00.731 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:29.777 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:06:00.757 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:29.778 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:06:00.737 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:29.782 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:06:00.741 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:29.788 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:06:00.769 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:29.812 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:06:00.771 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:29.833 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:06:00.814 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:31.340 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:31.462 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.461+0000 7feaf1364640 10 monclient: get_monmap_and_config 2024-03-25T18:09:31.462 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.461+0000 7feaf1364640 10 monclient: build_initial_monmap 2024-03-25T18:09:31.462 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.461+0000 7feaf1364640 10 monclient: monmap: 2024-03-25T18:09:31.462 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:09:31.463 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:31.463 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:09:31.462197+0000 2024-03-25T18:09:31.463 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:09:31.462197+0000 2024-03-25T18:09:31.463 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:31.463 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:31.463 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:09:31.463 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:09:31.463 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:09:31.464 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:31.466 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.462+0000 7feaf1364640 1 Processor -- start 2024-03-25T18:09:31.466 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.462+0000 7feaf1364640 1 -- start start 2024-03-25T18:09:31.466 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.462+0000 7feaf1364640 10 monclient: init 2024-03-25T18:09:31.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.463+0000 7feaf1364640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:31.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.463+0000 7feaf1364640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:09:31.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.463+0000 7feaf1364640 10 monclient(hunting): picked mon.noname-c con 0x7feaec0b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T18:09:31.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.463+0000 7feaf1364640 10 monclient(hunting): picked mon.noname-b con 0x7feaec0b1730 addr v1:172.21.15.66:6790/0 2024-03-25T18:09:31.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.463+0000 7feaf1364640 10 monclient(hunting): picked mon.noname-a con 0x7feaec14be90 addr v1:172.21.15.66:6789/0 2024-03-25T18:09:31.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.463+0000 7feaf1364640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaec0af940 con 0x7feaec14be90 2024-03-25T18:09:31.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.463+0000 7feaf1364640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaec0b0a70 con 0x7feaec0b4d60 2024-03-25T18:09:31.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.463+0000 7feaf1364640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaec0b0d50 con 0x7feaec0b1730 2024-03-25T18:09:31.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.463+0000 7feaf1364640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:31.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.463+0000 7feaf1364640 10 monclient(hunting): authenticate will time out at 7320.522616s 2024-03-25T18:09:31.469 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.469+0000 7feaeaffd640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7feaec0b4d60 0x7feaec13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:51726/0 (socket says 172.21.15.66:51726) 2024-03-25T18:09:31.469 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.469+0000 7feaeaffd640 1 -- 172.21.15.66:0/1014365309 learned_addr learned my addr 172.21.15.66:0/1014365309 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:09:31.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.486+0000 7feaea7fc640 1 -- 172.21.15.66:0/1014365309 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1860313652 0 0) 0x7feaec0af940 con 0x7feaec14be90 2024-03-25T18:09:31.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.486+0000 7feaea7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:31.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.486+0000 7feaea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:31.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.486+0000 7feaea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:31.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.486+0000 7feaea7fc640 10 monclient(hunting): my global_id is 4476 2024-03-25T18:09:31.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.486+0000 7feaea7fc640 1 -- 172.21.15.66:0/1014365309 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fead8003520 con 0x7feaec14be90 2024-03-25T18:09:31.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.486+0000 7feaea7fc640 1 -- 172.21.15.66:0/1014365309 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1681431861 0 0) 0x7feaec0b0a70 con 0x7feaec0b4d60 2024-03-25T18:09:31.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.486+0000 7feaea7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:31.488 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.486+0000 7feaea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:31.488 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.486+0000 7feaea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:31.488 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.486+0000 7feaea7fc640 10 monclient(hunting): my global_id is 4468 2024-03-25T18:09:31.488 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.486+0000 7feaea7fc640 1 -- 172.21.15.66:0/1014365309 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaec0af940 con 0x7feaec0b4d60 2024-03-25T18:09:31.506 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.506+0000 7feaea7fc640 1 -- 172.21.15.66:0/1014365309 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3598166165 0 0) 0x7fead8003520 con 0x7feaec14be90 2024-03-25T18:09:31.506 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.506+0000 7feaea7fc640 1 -- 172.21.15.66:0/1014365309 >> v1:172.21.15.66:6790/0 conn(0x7feaec0b1730 legacy=0x7feaec0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:31.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.506+0000 7feaea7fc640 1 -- 172.21.15.66:0/1014365309 >> v1:172.21.15.193:6789/0 conn(0x7feaec0b4d60 legacy=0x7feaec13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:31.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.506+0000 7feaea7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:09:31.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.506+0000 7feaea7fc640 1 monclient: found mon.noname-a 2024-03-25T18:09:31.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.506+0000 7feaea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:31.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.506+0000 7feaea7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.66:6789/0 2024-03-25T18:09:31.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.506+0000 7feaea7fc640 1 -- 172.21.15.66:0/1014365309 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaec0b1030 con 0x7feaec14be90 2024-03-25T18:09:31.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.506+0000 7feaea7fc640 10 monclient: _finish_auth 0 2024-03-25T18:09:31.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.506+0000 7feaea7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:09:31.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.506+0000 7feaea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:31.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.506+0000 7feaea7fc640 1 -- 172.21.15.66:0/1014365309 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7feae4001a10 con 0x7feaec14be90 2024-03-25T18:09:31.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.506+0000 7feaea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:31.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.506+0000 7feaea7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:09:31.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.506+0000 7feaea7fc640 10 monclient: dump: 2024-03-25T18:09:31.507 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:31.508 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:31.508 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:31.508 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:31.508 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:31.508 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:31.508 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:31.508 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:31.508 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:31.508 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:31.508 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.506+0000 7feaf1364640 5 monclient: authenticate success, global_id 4476 2024-03-25T18:09:31.508 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.506+0000 7feaf1364640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:31.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.516+0000 7feaea7fc640 1 -- 172.21.15.66:0/1014365309 <== mon.0 v1:172.21.15.66:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feae4001b70 con 0x7feaec14be90 2024-03-25T18:09:31.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.516+0000 7feaea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:31.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.516+0000 7feaea7fc640 1 -- 172.21.15.66:0/1014365309 <== mon.0 v1:172.21.15.66:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7feae4001f80 con 0x7feaec14be90 2024-03-25T18:09:31.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.516+0000 7feaea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:31.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.516+0000 7feaea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:31.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.516+0000 7feaea7fc640 10 monclient: dump: 2024-03-25T18:09:31.518 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:31.518 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:31.518 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:31.518 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:31.518 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:31.518 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:31.518 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:31.519 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:31.519 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:31.519 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:31.519 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.516+0000 7feaf1364640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:31.519 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.516+0000 7feaf1364640 4 set_mon_vals no callback set 2024-03-25T18:09:31.523 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.517+0000 7feaf1364640 10 monclient: shutdown 2024-03-25T18:09:31.523 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.517+0000 7feaf1364640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:31.523 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.517+0000 7feaf1364640 1 -- 172.21.15.66:0/1014365309 >> v1:172.21.15.66:6789/0 conn(0x7feaec14be90 legacy=0x7feaec14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:31.524 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.517+0000 7feaf1364640 1 -- 172.21.15.66:0/1014365309 shutdown_connections 2024-03-25T18:09:31.524 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.517+0000 7feaf1364640 1 -- 172.21.15.66:0/1014365309 >> 172.21.15.66:0/1014365309 conn(0x7feaec0617a0 msgr2=0x7feaec061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:31.524 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.517+0000 7feaf1364640 1 -- 172.21.15.66:0/1014365309 shutdown_connections 2024-03-25T18:09:31.524 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.517+0000 7feaf1364640 1 -- 172.21.15.66:0/1014365309 wait complete. 2024-03-25T18:09:31.524 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.518+0000 7feaf1364640 10 monclient: build_initial_monmap 2024-03-25T18:09:31.524 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.518+0000 7feaf1364640 10 monclient: monmap: 2024-03-25T18:09:31.524 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:09:31.525 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:31.525 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:09:31.525 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:09:31.525 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:31.525 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:09:31.525 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:09:31.526 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:09:31.526 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:09:31.526 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:31.526 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.518+0000 7feaf1364640 1 librados: starting msgr at 2024-03-25T18:09:31.526 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.518+0000 7feaf1364640 1 librados: starting objecter 2024-03-25T18:09:31.526 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.518+0000 7feaf1364640 1 Processor -- start 2024-03-25T18:09:31.527 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.518+0000 7feaf1364640 1 -- start start 2024-03-25T18:09:31.527 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.518+0000 7feaf1364640 1 librados: setting wanted keys 2024-03-25T18:09:31.527 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.518+0000 7feaf1364640 1 librados: calling monclient init 2024-03-25T18:09:31.527 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.518+0000 7feaf1364640 10 monclient: init 2024-03-25T18:09:31.527 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.519+0000 7feaf1364640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:31.527 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.519+0000 7feaf1364640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:09:31.527 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.519+0000 7feaf1364640 10 monclient(hunting): picked mon.noname-c con 0x7feaec0b1730 addr v1:172.21.15.66:6790/0 2024-03-25T18:09:31.528 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.519+0000 7feaf1364640 10 monclient(hunting): picked mon.noname-b con 0x7feaec0b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T18:09:31.528 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.519+0000 7feaf1364640 10 monclient(hunting): picked mon.noname-a con 0x7feaec14be90 addr v1:172.21.15.66:6789/0 2024-03-25T18:09:31.528 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.519+0000 7feaf1364640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaec134a00 con 0x7feaec14be90 2024-03-25T18:09:31.528 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.519+0000 7feaf1364640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaec139050 con 0x7feaec0b4d60 2024-03-25T18:09:31.528 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.519+0000 7feaf1364640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaec13a330 con 0x7feaec0b1730 2024-03-25T18:09:31.528 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.519+0000 7feaf1364640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:31.528 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.519+0000 7feaf1364640 10 monclient(hunting): authenticate will time out at 7320.578551s 2024-03-25T18:09:31.529 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.524+0000 7feae3fff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7feaec0b4d60 0x7feaec133be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:51748/0 (socket says 172.21.15.66:51748) 2024-03-25T18:09:31.529 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.524+0000 7feae3fff640 1 -- 172.21.15.66:0/3380904235 learned_addr learned my addr 172.21.15.66:0/3380904235 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:09:31.532 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.531+0000 7feae37fe640 10 client.?.objecter ms_handle_connect 0x7feaec0b4d60 2024-03-25T18:09:31.532 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.531+0000 7feae37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:31.540 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.539+0000 7feae37fe640 10 client.?.objecter ms_handle_connect 0x7feaec0b1730 2024-03-25T18:09:31.540 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.539+0000 7feae37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:31.543 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.542+0000 7feae37fe640 10 client.?.objecter ms_handle_connect 0x7feaec14be90 2024-03-25T18:09:31.543 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.542+0000 7feae37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:31.548 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.547+0000 7feae37fe640 1 -- 172.21.15.66:0/3380904235 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3114272153 0 0) 0x7feaec139050 con 0x7feaec0b4d60 2024-03-25T18:09:31.548 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.547+0000 7feae37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:31.548 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.547+0000 7feae37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:31.548 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.547+0000 7feae37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:31.548 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.547+0000 7feae37fe640 10 monclient(hunting): my global_id is 4471 2024-03-25T18:09:31.548 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.547+0000 7feae37fe640 1 -- 172.21.15.66:0/3380904235 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feacc004eb0 con 0x7feaec0b4d60 2024-03-25T18:09:31.550 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.549+0000 7feae37fe640 1 -- 172.21.15.66:0/3380904235 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842934245 0 0) 0x7feaec134a00 con 0x7feaec14be90 2024-03-25T18:09:31.550 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.549+0000 7feae37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:31.550 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.549+0000 7feae37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:31.550 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.549+0000 7feae37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:31.550 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.549+0000 7feae37fe640 10 monclient(hunting): my global_id is 4479 2024-03-25T18:09:31.550 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.549+0000 7feae37fe640 1 -- 172.21.15.66:0/3380904235 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaec139050 con 0x7feaec14be90 2024-03-25T18:09:31.556 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.555+0000 7feae37fe640 1 -- 172.21.15.66:0/3380904235 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2712119880 0 0) 0x7feaec13a330 con 0x7feaec0b1730 2024-03-25T18:09:31.556 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.555+0000 7feae37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:31.556 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.555+0000 7feae37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:31.556 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.555+0000 7feae37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:31.556 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.555+0000 7feae37fe640 10 monclient(hunting): my global_id is 4472 2024-03-25T18:09:31.556 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.555+0000 7feae37fe640 1 -- 172.21.15.66:0/3380904235 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaec134a00 con 0x7feaec0b1730 2024-03-25T18:09:31.558 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.557+0000 7feae37fe640 1 -- 172.21.15.66:0/3380904235 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1829595191 0 0) 0x7feacc004eb0 con 0x7feaec0b4d60 2024-03-25T18:09:31.558 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.557+0000 7feae37fe640 1 -- 172.21.15.66:0/3380904235 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feaec13a330 con 0x7feaec0b4d60 2024-03-25T18:09:31.558 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.557+0000 7feae37fe640 1 -- 172.21.15.66:0/3380904235 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fead4001a30 con 0x7feaec0b4d60 2024-03-25T18:09:31.558 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.557+0000 7feae37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:31.558 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.557+0000 7feae37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:31.558 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.557+0000 7feae37fe640 10 monclient(hunting): dump: 2024-03-25T18:09:31.558 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:31.558 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:31.558 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:31.558 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:31.558 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:31.559 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:31.559 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:31.559 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:31.559 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:31.559 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:31.559 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.557+0000 7feae37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:31.565 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.564+0000 7feae37fe640 1 -- 172.21.15.66:0/3380904235 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2092733334 0 0) 0x7feaec139050 con 0x7feaec14be90 2024-03-25T18:09:31.565 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.564+0000 7feae37fe640 1 -- 172.21.15.66:0/3380904235 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feacc004eb0 con 0x7feaec14be90 2024-03-25T18:09:31.565 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.564+0000 7feae37fe640 1 -- 172.21.15.66:0/3380904235 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7feae4001e80 con 0x7feaec14be90 2024-03-25T18:09:31.565 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.564+0000 7feae37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:31.565 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.564+0000 7feae37fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:31.565 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.564+0000 7feae37fe640 10 monclient(hunting): dump: 2024-03-25T18:09:31.565 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:31.565 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:31.565 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:31.565 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:31.565 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:31.565 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:31.565 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:31.566 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:31.566 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:31.566 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:31.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.571+0000 7feae37fe640 1 -- 172.21.15.66:0/3380904235 <== mon.0 v1:172.21.15.66:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4283173638 0 0) 0x7feacc004eb0 con 0x7feaec14be90 2024-03-25T18:09:31.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.571+0000 7feae37fe640 1 -- 172.21.15.66:0/3380904235 >> v1:172.21.15.66:6790/0 conn(0x7feaec0b1730 legacy=0x7feaec13ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:31.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.571+0000 7feae37fe640 1 -- 172.21.15.66:0/3380904235 >> v1:172.21.15.193:6789/0 conn(0x7feaec0b4d60 legacy=0x7feaec133be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:31.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.571+0000 7feae37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:09:31.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.571+0000 7feae37fe640 1 monclient: found mon.a 2024-03-25T18:09:31.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.571+0000 7feae37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:31.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.571+0000 7feae37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:09:31.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.571+0000 7feae37fe640 1 -- 172.21.15.66:0/3380904235 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaec13a610 con 0x7feaec14be90 2024-03-25T18:09:31.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.571+0000 7feae37fe640 10 monclient: _finish_auth 0 2024-03-25T18:09:31.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.571+0000 7feae37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:09:31.573 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.571+0000 7feae37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:31.573 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.571+0000 7feaf1364640 5 monclient: authenticate success, global_id 4479 2024-03-25T18:09:31.573 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.571+0000 7feaf1364640 10 monclient: _renew_subs 2024-03-25T18:09:31.573 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.571+0000 7feaf1364640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:09:31.573 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.571+0000 7feaf1364640 1 -- 172.21.15.66:0/3380904235 --> v1:172.21.15.66:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaec13a8f0 con 0x7feaec14be90 2024-03-25T18:09:31.573 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.571+0000 7feaf1364640 10 client.4479.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:31.573 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.571+0000 7feaf1364640 10 monclient: _renew_subs 2024-03-25T18:09:31.573 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.571+0000 7feaf1364640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:09:31.573 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.571+0000 7feaf1364640 1 -- 172.21.15.66:0/3380904235 --> v1:172.21.15.66:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feaec143140 con 0x7feaec14be90 2024-03-25T18:09:31.573 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.571+0000 7feaf1364640 1 librados: init done 2024-03-25T18:09:31.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.572+0000 7feaf1364640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:31.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.572+0000 7feaf1364640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:31.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.572+0000 7feaf1364640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:09:31.574 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.572+0000 7feaf1364640 1 -- 172.21.15.66:0/3380904235 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaec0627b0 con 0x7feaec14be90 2024-03-25T18:09:31.786 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.786+0000 7feae37fe640 1 -- 172.21.15.66:0/3380904235 <== mon.0 v1:172.21.15.66:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feae4001fe0 con 0x7feaec14be90 2024-03-25T18:09:31.786 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.786+0000 7feae37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:31.787 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.786+0000 7feae9ffb640 4 set_mon_vals no callback set 2024-03-25T18:09:31.879 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.879+0000 7feae37fe640 1 -- 172.21.15.66:0/3380904235 <== mon.0 v1:172.21.15.66:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7feae4003dc0 con 0x7feaec14be90 2024-03-25T18:09:31.879 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.879+0000 7feae37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:31.880 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.879+0000 7feae37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:31.880 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.879+0000 7feae37fe640 10 monclient: dump: 2024-03-25T18:09:31.880 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:31.880 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:31.880 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:31.880 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:31.880 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:31.880 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:31.880 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:31.880 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:31.880 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:31.880 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:31.881 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.880+0000 7feae37fe640 1 -- 172.21.15.66:0/3380904235 <== mon.0 v1:172.21.15.66:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7feae4019d90 con 0x7feaec14be90 2024-03-25T18:09:31.881 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.880+0000 7feaeaffd640 10 monclient: get_auth_request con 0x7feacc06ded0 auth_method 0 2024-03-25T18:09:31.881 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.881+0000 7feae37fe640 10 client.4479.objecter ms_handle_connect 0x7feacc06ded0 2024-03-25T18:09:31.886 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.885+0000 7feae37fe640 1 -- 172.21.15.66:0/3380904235 <== mon.0 v1:172.21.15.66:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7feae40538c0 con 0x7feaec14be90 2024-03-25T18:09:31.886 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.885+0000 7feae37fe640 10 client.4479.objecter ms_dispatch 0x7feaec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:31.886 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.885+0000 7feae37fe640 3 client.4479.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:31.886 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.885+0000 7feae37fe640 3 client.4479.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:31.887 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.886+0000 7feae37fe640 20 client.4479.objecter dump_active .. 0 homeless 2024-03-25T18:09:31.887 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.886+0000 7feae37fe640 1 -- 172.21.15.66:0/3380904235 <== mon.0 v1:172.21.15.66:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feae404f200 con 0x7feaec14be90 2024-03-25T18:09:31.887 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.886+0000 7feae37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:31.887 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:31.886+0000 7feae37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:32.140 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:32.139+0000 7feaf1364640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:32.140 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:32.139+0000 7feaf1364640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:32.140 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:32.139+0000 7feaf1364640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:09:32.140 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:32.139+0000 7feaf1364640 1 -- 172.21.15.66:0/3380904235 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feaec072730 con 0x7feaec14be90 2024-03-25T18:09:32.166 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:32.166+0000 7feae37fe640 1 -- 172.21.15.66:0/3380904235 <== mon.0 v1:172.21.15.66:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7feae4051300 con 0x7feaec14be90 2024-03-25T18:09:32.167 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:32.166+0000 7feae37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:32.167 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:32.166+0000 7feae37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:32.167 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:09:32.167 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:32.169 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:32.168+0000 7feaf1364640 10 librados: watch_flush enter 2024-03-25T18:09:32.169 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:32.168+0000 7feaf1364640 10 librados: watch_flush exit 2024-03-25T18:09:32.170 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:32.168+0000 7feaf1364640 20 client.4479.objecter shutdown clearing up homeless session... 2024-03-25T18:09:32.170 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:32.168+0000 7feaf1364640 10 client.4479.objecter successfully canceled tick 2024-03-25T18:09:32.170 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:32.168+0000 7feaf1364640 1 -- 172.21.15.66:0/3380904235 >> v1:172.21.15.193:6800/2275676803 conn(0x7feacc06ded0 legacy=0x7feacc070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:32.170 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:32.168+0000 7feaf1364640 10 monclient: shutdown 2024-03-25T18:09:32.170 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:32.168+0000 7feaf1364640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:32.170 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:32.168+0000 7feaf1364640 1 -- 172.21.15.66:0/3380904235 >> v1:172.21.15.66:6789/0 conn(0x7feaec14be90 legacy=0x7feaec1342f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:32.170 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:32.168+0000 7feaf1364640 1 -- 172.21.15.66:0/3380904235 shutdown_connections 2024-03-25T18:09:32.170 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:32.168+0000 7feaf1364640 1 -- 172.21.15.66:0/3380904235 >> 172.21.15.66:0/3380904235 conn(0x7feaec0617a0 msgr2=0x7feaec0b6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:32.170 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:32.168+0000 7feaf1364640 1 -- 172.21.15.66:0/3380904235 shutdown_connections 2024-03-25T18:09:32.170 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:32.168+0000 7feaf1364640 1 -- 172.21.15.66:0/3380904235 wait complete. 2024-03-25T18:09:32.170 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:32.168+0000 7feaf1364640 1 librados: shutdown 2024-03-25T18:09:32.181 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:38.987 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:39.112 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.111+0000 7ff251fcf640 10 monclient: get_monmap_and_config 2024-03-25T18:09:39.112 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.111+0000 7ff251fcf640 10 monclient: build_initial_monmap 2024-03-25T18:09:39.112 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.112+0000 7ff251fcf640 10 monclient: monmap: 2024-03-25T18:09:39.113 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:09:39.113 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:39.113 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:09:39.112500+0000 2024-03-25T18:09:39.113 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:09:39.112500+0000 2024-03-25T18:09:39.113 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:39.113 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:39.113 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:09:39.113 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:09:39.113 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:09:39.113 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:39.114 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.112+0000 7ff251fcf640 1 Processor -- start 2024-03-25T18:09:39.114 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.112+0000 7ff251fcf640 1 -- start start 2024-03-25T18:09:39.114 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.112+0000 7ff251fcf640 10 monclient: init 2024-03-25T18:09:39.115 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.113+0000 7ff251fcf640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:39.115 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.113+0000 7ff251fcf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:09:39.115 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.113+0000 7ff251fcf640 10 monclient(hunting): picked mon.noname-b con 0x7ff24c0a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:09:39.115 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.113+0000 7ff251fcf640 10 monclient(hunting): picked mon.noname-c con 0x7ff24c13f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:09:39.115 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.113+0000 7ff251fcf640 10 monclient(hunting): picked mon.noname-a con 0x7ff24c1430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:09:39.115 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.113+0000 7ff251fcf640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff24c142520 con 0x7ff24c1430c0 2024-03-25T18:09:39.115 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.113+0000 7ff251fcf640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff24c0b28a0 con 0x7ff24c13f710 2024-03-25T18:09:39.115 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.113+0000 7ff251fcf640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff24c0b2a30 con 0x7ff24c0a98f0 2024-03-25T18:09:39.115 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.113+0000 7ff251fcf640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:39.115 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.113+0000 7ff251fcf640 10 monclient(hunting): authenticate will time out at 7328.172838s 2024-03-25T18:09:39.125 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.125+0000 7ff24affd640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7ff24c13f710 0x7ff24c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:48748/0 (socket says 172.21.15.66:48748) 2024-03-25T18:09:39.126 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.125+0000 7ff24affd640 1 -- 172.21.15.66:0/1594911916 learned_addr learned my addr 172.21.15.66:0/1594911916 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:09:39.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.137+0000 7ff24a7fc640 1 -- 172.21.15.66:0/1594911916 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1938416857 0 0) 0x7ff24c0b2a30 con 0x7ff24c0a98f0 2024-03-25T18:09:39.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.137+0000 7ff24a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:39.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.137+0000 7ff24a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:39.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.137+0000 7ff24a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:39.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.137+0000 7ff24a7fc640 10 monclient(hunting): my global_id is 4475 2024-03-25T18:09:39.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.137+0000 7ff24a7fc640 1 -- 172.21.15.66:0/1594911916 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff234003520 con 0x7ff24c0a98f0 2024-03-25T18:09:39.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.137+0000 7ff24a7fc640 1 -- 172.21.15.66:0/1594911916 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590653683 0 0) 0x7ff24c0b28a0 con 0x7ff24c13f710 2024-03-25T18:09:39.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.137+0000 7ff24a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:39.139 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.137+0000 7ff24a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:39.139 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.137+0000 7ff24a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:39.139 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.137+0000 7ff24a7fc640 10 monclient(hunting): my global_id is 4474 2024-03-25T18:09:39.139 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.137+0000 7ff24a7fc640 1 -- 172.21.15.66:0/1594911916 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff24c0b2a30 con 0x7ff24c13f710 2024-03-25T18:09:39.139 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.139+0000 7ff24a7fc640 1 -- 172.21.15.66:0/1594911916 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 682805545 0 0) 0x7ff24c142520 con 0x7ff24c1430c0 2024-03-25T18:09:39.140 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.139+0000 7ff24a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:39.140 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.139+0000 7ff24a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:39.140 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.139+0000 7ff24a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:39.140 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.139+0000 7ff24a7fc640 10 monclient(hunting): my global_id is 4482 2024-03-25T18:09:39.140 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.139+0000 7ff24a7fc640 1 -- 172.21.15.66:0/1594911916 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff24c0b28a0 con 0x7ff24c1430c0 2024-03-25T18:09:39.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.147+0000 7ff24a7fc640 1 -- 172.21.15.66:0/1594911916 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4030971073 0 0) 0x7ff24c0b28a0 con 0x7ff24c1430c0 2024-03-25T18:09:39.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.147+0000 7ff24a7fc640 1 -- 172.21.15.66:0/1594911916 >> v1:172.21.15.66:6790/0 conn(0x7ff24c0a98f0 legacy=0x7ff24c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:39.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.147+0000 7ff24a7fc640 1 -- 172.21.15.66:0/1594911916 >> v1:172.21.15.193:6789/0 conn(0x7ff24c13f710 legacy=0x7ff24c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:39.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.147+0000 7ff24a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:09:39.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.147+0000 7ff24a7fc640 1 monclient: found mon.noname-a 2024-03-25T18:09:39.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.147+0000 7ff24a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:39.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.147+0000 7ff24a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.66:6789/0 2024-03-25T18:09:39.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.147+0000 7ff24a7fc640 1 -- 172.21.15.66:0/1594911916 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24c0b44c0 con 0x7ff24c1430c0 2024-03-25T18:09:39.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.147+0000 7ff24a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:09:39.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.147+0000 7ff24a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:09:39.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.147+0000 7ff24a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:39.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.147+0000 7ff24a7fc640 1 -- 172.21.15.66:0/1594911916 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7ff244001930 con 0x7ff24c1430c0 2024-03-25T18:09:39.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.147+0000 7ff251fcf640 5 monclient: authenticate success, global_id 4482 2024-03-25T18:09:39.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.147+0000 7ff251fcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:39.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.147+0000 7ff24a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:39.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.147+0000 7ff24a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:09:39.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.147+0000 7ff24a7fc640 10 monclient: dump: 2024-03-25T18:09:39.149 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:39.149 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:39.149 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:39.149 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:39.149 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:39.149 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:39.149 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:39.149 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:39.150 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:39.150 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:39.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.147+0000 7ff251fcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:39.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.157+0000 7ff24a7fc640 1 -- 172.21.15.66:0/1594911916 <== mon.0 v1:172.21.15.66:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff244001a90 con 0x7ff24c1430c0 2024-03-25T18:09:39.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.157+0000 7ff24a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:39.158 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.157+0000 7ff24a7fc640 1 -- 172.21.15.66:0/1594911916 <== mon.0 v1:172.21.15.66:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7ff244001ee0 con 0x7ff24c1430c0 2024-03-25T18:09:39.158 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.157+0000 7ff24a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:39.158 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.157+0000 7ff24a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:39.158 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.157+0000 7ff24a7fc640 10 monclient: dump: 2024-03-25T18:09:39.158 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:39.158 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:39.158 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:39.158 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:39.158 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:39.158 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:39.158 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:39.158 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:39.158 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:39.158 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:39.159 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.157+0000 7ff251fcf640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:39.159 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.157+0000 7ff251fcf640 4 set_mon_vals no callback set 2024-03-25T18:09:39.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.158+0000 7ff251fcf640 10 monclient: shutdown 2024-03-25T18:09:39.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.158+0000 7ff251fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:39.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.158+0000 7ff251fcf640 1 -- 172.21.15.66:0/1594911916 >> v1:172.21.15.66:6789/0 conn(0x7ff24c1430c0 legacy=0x7ff24c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:39.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.158+0000 7ff251fcf640 1 -- 172.21.15.66:0/1594911916 shutdown_connections 2024-03-25T18:09:39.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.158+0000 7ff251fcf640 1 -- 172.21.15.66:0/1594911916 >> 172.21.15.66:0/1594911916 conn(0x7ff24c0b7ea0 msgr2=0x7ff24c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:39.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.158+0000 7ff251fcf640 1 -- 172.21.15.66:0/1594911916 shutdown_connections 2024-03-25T18:09:39.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.158+0000 7ff251fcf640 1 -- 172.21.15.66:0/1594911916 wait complete. 2024-03-25T18:09:39.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.159+0000 7ff251fcf640 10 monclient: build_initial_monmap 2024-03-25T18:09:39.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.159+0000 7ff251fcf640 10 monclient: monmap: 2024-03-25T18:09:39.161 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:09:39.161 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:39.161 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:09:39.162 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:09:39.162 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:39.162 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:09:39.162 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:09:39.162 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:09:39.162 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:09:39.162 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:39.162 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.159+0000 7ff251fcf640 1 librados: starting msgr at 2024-03-25T18:09:39.162 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.159+0000 7ff251fcf640 1 librados: starting objecter 2024-03-25T18:09:39.162 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.159+0000 7ff251fcf640 1 Processor -- start 2024-03-25T18:09:39.162 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.159+0000 7ff251fcf640 1 -- start start 2024-03-25T18:09:39.162 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.159+0000 7ff251fcf640 1 librados: setting wanted keys 2024-03-25T18:09:39.162 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.159+0000 7ff251fcf640 1 librados: calling monclient init 2024-03-25T18:09:39.162 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.159+0000 7ff251fcf640 10 monclient: init 2024-03-25T18:09:39.162 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.160+0000 7ff251fcf640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:39.163 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.160+0000 7ff251fcf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:09:39.163 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.160+0000 7ff251fcf640 10 monclient(hunting): picked mon.noname-b con 0x7ff24c0a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:09:39.163 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.160+0000 7ff251fcf640 10 monclient(hunting): picked mon.noname-a con 0x7ff24c13f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:09:39.163 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.160+0000 7ff251fcf640 10 monclient(hunting): picked mon.noname-c con 0x7ff24c1430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:09:39.163 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.160+0000 7ff251fcf640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff24c1397c0 con 0x7ff24c13f710 2024-03-25T18:09:39.163 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.160+0000 7ff251fcf640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff24c1d8260 con 0x7ff24c0a98f0 2024-03-25T18:09:39.163 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.160+0000 7ff251fcf640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff24c138ab0 con 0x7ff24c1430c0 2024-03-25T18:09:39.163 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.160+0000 7ff251fcf640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:39.163 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.160+0000 7ff251fcf640 10 monclient(hunting): authenticate will time out at 7328.219704s 2024-03-25T18:09:39.174 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.173+0000 7ff24b7fe640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7ff24c0a98f0 0x7ff24c1d7b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:48788/0 (socket says 172.21.15.66:48788) 2024-03-25T18:09:39.174 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.173+0000 7ff24b7fe640 1 -- 172.21.15.66:0/1777682547 learned_addr learned my addr 172.21.15.66:0/1777682547 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:09:39.180 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.180+0000 7ff233fff640 10 client.?.objecter ms_handle_connect 0x7ff24c0a98f0 2024-03-25T18:09:39.180 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.180+0000 7ff233fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:39.183 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.183+0000 7ff233fff640 10 client.?.objecter ms_handle_connect 0x7ff24c13f710 2024-03-25T18:09:39.183 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.183+0000 7ff233fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:39.193 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.192+0000 7ff233fff640 1 -- 172.21.15.66:0/1777682547 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1858237353 0 0) 0x7ff24c1d8260 con 0x7ff24c0a98f0 2024-03-25T18:09:39.193 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.192+0000 7ff233fff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:39.193 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.192+0000 7ff233fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:39.193 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.192+0000 7ff233fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:39.193 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.192+0000 7ff233fff640 10 monclient(hunting): my global_id is 4477 2024-03-25T18:09:39.193 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.192+0000 7ff233fff640 1 -- 172.21.15.66:0/1777682547 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff238004eb0 con 0x7ff24c0a98f0 2024-03-25T18:09:39.194 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.193+0000 7ff233fff640 10 client.?.objecter ms_handle_connect 0x7ff24c1430c0 2024-03-25T18:09:39.194 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.193+0000 7ff233fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:39.204 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.204+0000 7ff233fff640 1 -- 172.21.15.66:0/1777682547 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1550805644 0 0) 0x7ff24c1397c0 con 0x7ff24c13f710 2024-03-25T18:09:39.205 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.204+0000 7ff233fff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:39.205 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.204+0000 7ff233fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:39.205 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.204+0000 7ff233fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:39.205 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.204+0000 7ff233fff640 10 monclient(hunting): my global_id is 4485 2024-03-25T18:09:39.205 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.204+0000 7ff233fff640 1 -- 172.21.15.66:0/1777682547 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff24c1d8260 con 0x7ff24c13f710 2024-03-25T18:09:39.209 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.209+0000 7ff233fff640 1 -- 172.21.15.66:0/1777682547 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697497216 0 0) 0x7ff24c138ab0 con 0x7ff24c1430c0 2024-03-25T18:09:39.210 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.209+0000 7ff233fff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:39.210 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.209+0000 7ff233fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:39.210 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.209+0000 7ff233fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:39.210 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.209+0000 7ff233fff640 10 monclient(hunting): my global_id is 4478 2024-03-25T18:09:39.210 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.209+0000 7ff233fff640 1 -- 172.21.15.66:0/1777682547 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff24c1397c0 con 0x7ff24c1430c0 2024-03-25T18:09:39.219 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.218+0000 7ff233fff640 1 -- 172.21.15.66:0/1777682547 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3409513493 0 0) 0x7ff238004eb0 con 0x7ff24c0a98f0 2024-03-25T18:09:39.219 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.218+0000 7ff233fff640 1 -- 172.21.15.66:0/1777682547 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff24c138ab0 con 0x7ff24c0a98f0 2024-03-25T18:09:39.219 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.218+0000 7ff233fff640 1 -- 172.21.15.66:0/1777682547 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7ff23c0019b0 con 0x7ff24c0a98f0 2024-03-25T18:09:39.219 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.218+0000 7ff233fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:39.219 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.218+0000 7ff233fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:39.219 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.218+0000 7ff233fff640 10 monclient(hunting): dump: 2024-03-25T18:09:39.219 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:39.219 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:39.219 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:39.219 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:39.219 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:39.220 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:39.220 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:39.220 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:39.220 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:39.220 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:39.220 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.218+0000 7ff233fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:39.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.232+0000 7ff233fff640 1 -- 172.21.15.66:0/1777682547 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4108447023 0 0) 0x7ff24c138ab0 con 0x7ff24c0a98f0 2024-03-25T18:09:39.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.232+0000 7ff233fff640 1 -- 172.21.15.66:0/1777682547 >> v1:172.21.15.66:6790/0 conn(0x7ff24c1430c0 legacy=0x7ff24c134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:39.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.232+0000 7ff233fff640 1 -- 172.21.15.66:0/1777682547 >> v1:172.21.15.66:6789/0 conn(0x7ff24c13f710 legacy=0x7ff24c1383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:39.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.232+0000 7ff233fff640 10 monclient: _finish_hunting 0 2024-03-25T18:09:39.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.232+0000 7ff233fff640 1 monclient: found mon.b 2024-03-25T18:09:39.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.232+0000 7ff233fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:39.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.232+0000 7ff233fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:09:39.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.232+0000 7ff233fff640 1 -- 172.21.15.66:0/1777682547 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24c138c40 con 0x7ff24c0a98f0 2024-03-25T18:09:39.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.232+0000 7ff233fff640 10 monclient: _finish_auth 0 2024-03-25T18:09:39.234 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.232+0000 7ff233fff640 10 monclient: _check_auth_tickets 2024-03-25T18:09:39.234 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.232+0000 7ff233fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:39.234 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.232+0000 7ff251fcf640 5 monclient: authenticate success, global_id 4477 2024-03-25T18:09:39.234 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.232+0000 7ff251fcf640 10 monclient: _renew_subs 2024-03-25T18:09:39.234 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.232+0000 7ff251fcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:09:39.234 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.232+0000 7ff251fcf640 1 -- 172.21.15.66:0/1777682547 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff24c137690 con 0x7ff24c0a98f0 2024-03-25T18:09:39.234 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.232+0000 7ff251fcf640 10 client.4477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:39.234 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.232+0000 7ff251fcf640 10 monclient: _renew_subs 2024-03-25T18:09:39.234 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.232+0000 7ff251fcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:09:39.234 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.232+0000 7ff251fcf640 1 -- 172.21.15.66:0/1777682547 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff24c134900 con 0x7ff24c0a98f0 2024-03-25T18:09:39.234 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.232+0000 7ff251fcf640 1 librados: init done 2024-03-25T18:09:39.235 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.233+0000 7ff251fcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:39.235 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.233+0000 7ff251fcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:39.235 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.233+0000 7ff251fcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:09:39.235 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.233+0000 7ff251fcf640 1 -- 172.21.15.66:0/1777682547 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff24c1d79a0 con 0x7ff24c0a98f0 2024-03-25T18:09:39.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.499+0000 7ff233fff640 1 -- 172.21.15.66:0/1777682547 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff23c0035f0 con 0x7ff24c0a98f0 2024-03-25T18:09:39.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.499+0000 7ff233fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:39.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.499+0000 7ff233fff640 1 -- 172.21.15.66:0/1777682547 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7ff23c003980 con 0x7ff24c0a98f0 2024-03-25T18:09:39.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.499+0000 7ff233fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:39.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.499+0000 7ff233fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:39.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.499+0000 7ff249ffb640 4 set_mon_vals no callback set 2024-03-25T18:09:39.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.499+0000 7ff233fff640 10 monclient: dump: 2024-03-25T18:09:39.500 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:39.500 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:39.500 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:39.500 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:39.500 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:39.501 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:39.501 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:39.501 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:39.501 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:39.501 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:39.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.499+0000 7ff233fff640 1 -- 172.21.15.66:0/1777682547 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7ff23c003ba0 con 0x7ff24c0a98f0 2024-03-25T18:09:39.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.500+0000 7ff24affd640 10 monclient: get_auth_request con 0x7ff23806de20 auth_method 0 2024-03-25T18:09:39.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.500+0000 7ff233fff640 10 client.4477.objecter ms_handle_connect 0x7ff23806de20 2024-03-25T18:09:39.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.506+0000 7ff233fff640 1 -- 172.21.15.66:0/1777682547 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7ff23c054830 con 0x7ff24c0a98f0 2024-03-25T18:09:39.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.506+0000 7ff233fff640 10 client.4477.objecter ms_dispatch 0x7ff24c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:39.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.506+0000 7ff233fff640 3 client.4477.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:39.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.506+0000 7ff233fff640 3 client.4477.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:39.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.506+0000 7ff233fff640 20 client.4477.objecter dump_active .. 0 homeless 2024-03-25T18:09:39.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.506+0000 7ff233fff640 1 -- 172.21.15.66:0/1777682547 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff23c050340 con 0x7ff24c0a98f0 2024-03-25T18:09:39.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.506+0000 7ff233fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:39.507 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.506+0000 7ff233fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:39.728 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.727+0000 7ff251fcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:39.728 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.727+0000 7ff251fcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:39.728 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.728+0000 7ff251fcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:09:39.728 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.728+0000 7ff251fcf640 1 -- 172.21.15.66:0/1777682547 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff24c0b3ef0 con 0x7ff24c0a98f0 2024-03-25T18:09:39.752 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.752+0000 7ff233fff640 1 -- 172.21.15.66:0/1777682547 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7ff23c052270 con 0x7ff24c0a98f0 2024-03-25T18:09:39.753 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.752+0000 7ff233fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:39.753 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.752+0000 7ff233fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:39.753 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:09:39.753 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:39.755 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.754+0000 7ff251fcf640 10 librados: watch_flush enter 2024-03-25T18:09:39.756 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.754+0000 7ff251fcf640 10 librados: watch_flush exit 2024-03-25T18:09:39.756 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.754+0000 7ff251fcf640 20 client.4477.objecter shutdown clearing up homeless session... 2024-03-25T18:09:39.756 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.754+0000 7ff251fcf640 10 client.4477.objecter successfully canceled tick 2024-03-25T18:09:39.756 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.754+0000 7ff251fcf640 1 -- 172.21.15.66:0/1777682547 >> v1:172.21.15.193:6800/2275676803 conn(0x7ff23806de20 legacy=0x7ff2380702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:39.756 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.755+0000 7ff251fcf640 10 monclient: shutdown 2024-03-25T18:09:39.756 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.755+0000 7ff251fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:39.756 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.755+0000 7ff251fcf640 1 -- 172.21.15.66:0/1777682547 >> v1:172.21.15.193:6789/0 conn(0x7ff24c0a98f0 legacy=0x7ff24c1d7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:39.756 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.755+0000 7ff251fcf640 1 -- 172.21.15.66:0/1777682547 shutdown_connections 2024-03-25T18:09:39.756 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.755+0000 7ff251fcf640 1 -- 172.21.15.66:0/1777682547 >> 172.21.15.66:0/1777682547 conn(0x7ff24c0b7ea0 msgr2=0x7ff24c13d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:39.756 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.755+0000 7ff251fcf640 1 -- 172.21.15.66:0/1777682547 shutdown_connections 2024-03-25T18:09:39.756 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.755+0000 7ff251fcf640 1 -- 172.21.15.66:0/1777682547 wait complete. 2024-03-25T18:09:39.756 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:39.755+0000 7ff251fcf640 1 librados: shutdown 2024-03-25T18:09:39.768 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:46.574 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:46.694 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.693+0000 7fd61d61e640 10 monclient: get_monmap_and_config 2024-03-25T18:09:46.694 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.693+0000 7fd61d61e640 10 monclient: build_initial_monmap 2024-03-25T18:09:46.694 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.693+0000 7fd61d61e640 10 monclient: monmap: 2024-03-25T18:09:46.694 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:09:46.694 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:46.694 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:09:46.693858+0000 2024-03-25T18:09:46.694 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:09:46.693858+0000 2024-03-25T18:09:46.694 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:46.694 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:46.695 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:09:46.695 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:09:46.695 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:09:46.695 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:46.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.694+0000 7fd61d61e640 1 Processor -- start 2024-03-25T18:09:46.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.694+0000 7fd61d61e640 1 -- start start 2024-03-25T18:09:46.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.694+0000 7fd61d61e640 10 monclient: init 2024-03-25T18:09:46.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.695+0000 7fd61d61e640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:46.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.695+0000 7fd61d61e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:09:46.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.695+0000 7fd61d61e640 10 monclient(hunting): picked mon.noname-a con 0x7fd6180a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:09:46.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.695+0000 7fd61d61e640 10 monclient(hunting): picked mon.noname-b con 0x7fd61813f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:09:46.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.695+0000 7fd61d61e640 10 monclient(hunting): picked mon.noname-c con 0x7fd6181430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:09:46.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.695+0000 7fd61d61e640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd618142520 con 0x7fd6180a98f0 2024-03-25T18:09:46.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.695+0000 7fd61d61e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6180b28a0 con 0x7fd6181430c0 2024-03-25T18:09:46.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.695+0000 7fd61d61e640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6180b2a30 con 0x7fd61813f710 2024-03-25T18:09:46.697 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.695+0000 7fd61d61e640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:46.697 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.695+0000 7fd61d61e640 10 monclient(hunting): authenticate will time out at 7335.754407s 2024-03-25T18:09:46.700 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.699+0000 7fd6167fc640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7fd61813f710 0x7fd6181429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:49306/0 (socket says 172.21.15.66:49306) 2024-03-25T18:09:46.700 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.699+0000 7fd6167fc640 1 -- 172.21.15.66:0/3866086402 learned_addr learned my addr 172.21.15.66:0/3866086402 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:09:46.709 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.708+0000 7fd615ffb640 1 -- 172.21.15.66:0/3866086402 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2004798603 0 0) 0x7fd6180b2a30 con 0x7fd61813f710 2024-03-25T18:09:46.709 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.708+0000 7fd615ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:09:46.709 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.708+0000 7fd615ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:46.709 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.708+0000 7fd615ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:46.709 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.708+0000 7fd615ffb640 10 monclient(hunting): my global_id is 4481 2024-03-25T18:09:46.709 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.708+0000 7fd615ffb640 1 -- 172.21.15.66:0/3866086402 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd600003520 con 0x7fd61813f710 2024-03-25T18:09:46.720 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.719+0000 7fd615ffb640 1 -- 172.21.15.66:0/3866086402 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 637918706 0 0) 0x7fd600003520 con 0x7fd61813f710 2024-03-25T18:09:46.720 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.719+0000 7fd615ffb640 1 -- 172.21.15.66:0/3866086402 >> v1:172.21.15.193:6789/0 conn(0x7fd6181430c0 legacy=0x7fd61814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:46.720 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.719+0000 7fd615ffb640 1 -- 172.21.15.66:0/3866086402 >> v1:172.21.15.66:6789/0 conn(0x7fd6180a98f0 legacy=0x7fd6180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:46.720 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.719+0000 7fd615ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:09:46.720 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.719+0000 7fd615ffb640 1 monclient: found mon.noname-b 2024-03-25T18:09:46.720 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.719+0000 7fd615ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:46.720 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.719+0000 7fd615ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.66:6790/0 2024-03-25T18:09:46.720 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.719+0000 7fd615ffb640 1 -- 172.21.15.66:0/3866086402 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6180b44c0 con 0x7fd61813f710 2024-03-25T18:09:46.720 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.719+0000 7fd615ffb640 10 monclient: _finish_auth 0 2024-03-25T18:09:46.720 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.719+0000 7fd615ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:09:46.721 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.719+0000 7fd615ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:46.721 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.719+0000 7fd615ffb640 1 -- 172.21.15.66:0/3866086402 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fd60c001690 con 0x7fd61813f710 2024-03-25T18:09:46.721 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.719+0000 7fd615ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:46.721 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.719+0000 7fd615ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:46.721 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.719+0000 7fd615ffb640 10 monclient: dump: 2024-03-25T18:09:46.721 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:46.721 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:46.721 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:46.721 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:46.721 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:46.721 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:46.721 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:46.721 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:46.721 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:46.721 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:46.721 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.719+0000 7fd61d61e640 5 monclient: authenticate success, global_id 4481 2024-03-25T18:09:46.722 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.719+0000 7fd61d61e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:46.731 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.730+0000 7fd615ffb640 1 -- 172.21.15.66:0/3866086402 <== mon.2 v1:172.21.15.66:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd60c0017f0 con 0x7fd61813f710 2024-03-25T18:09:46.731 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.730+0000 7fd615ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:46.731 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.730+0000 7fd615ffb640 1 -- 172.21.15.66:0/3866086402 <== mon.2 v1:172.21.15.66:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fd60c001c00 con 0x7fd61813f710 2024-03-25T18:09:46.731 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.730+0000 7fd61d61e640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:46.731 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.730+0000 7fd61d61e640 4 set_mon_vals no callback set 2024-03-25T18:09:46.731 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.730+0000 7fd615ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:46.731 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.730+0000 7fd615ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:46.731 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.730+0000 7fd615ffb640 10 monclient: dump: 2024-03-25T18:09:46.731 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:46.731 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:46.731 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:46.732 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:46.732 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:46.732 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:46.732 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:46.732 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:46.732 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:46.732 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:46.735 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.731+0000 7fd61d61e640 10 monclient: shutdown 2024-03-25T18:09:46.735 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.731+0000 7fd61d61e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:46.735 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.731+0000 7fd61d61e640 1 -- 172.21.15.66:0/3866086402 >> v1:172.21.15.66:6790/0 conn(0x7fd61813f710 legacy=0x7fd6181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:46.735 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.731+0000 7fd61d61e640 1 -- 172.21.15.66:0/3866086402 shutdown_connections 2024-03-25T18:09:46.735 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.731+0000 7fd61d61e640 1 -- 172.21.15.66:0/3866086402 >> 172.21.15.66:0/3866086402 conn(0x7fd6180b7ea0 msgr2=0x7fd6180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:46.735 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.731+0000 7fd61d61e640 1 -- 172.21.15.66:0/3866086402 shutdown_connections 2024-03-25T18:09:46.735 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.731+0000 7fd61d61e640 1 -- 172.21.15.66:0/3866086402 wait complete. 2024-03-25T18:09:46.735 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.732+0000 7fd61d61e640 10 monclient: build_initial_monmap 2024-03-25T18:09:46.736 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.732+0000 7fd61d61e640 10 monclient: monmap: 2024-03-25T18:09:46.736 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:09:46.736 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:46.736 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:09:46.736 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:09:46.736 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:46.736 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:09:46.736 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:09:46.736 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:09:46.736 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:09:46.736 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:46.736 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.732+0000 7fd61d61e640 1 librados: starting msgr at 2024-03-25T18:09:46.736 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.732+0000 7fd61d61e640 1 librados: starting objecter 2024-03-25T18:09:46.736 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.733+0000 7fd61d61e640 1 Processor -- start 2024-03-25T18:09:46.736 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.733+0000 7fd61d61e640 1 -- start start 2024-03-25T18:09:46.736 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.733+0000 7fd61d61e640 1 librados: setting wanted keys 2024-03-25T18:09:46.737 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.733+0000 7fd61d61e640 1 librados: calling monclient init 2024-03-25T18:09:46.737 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.733+0000 7fd61d61e640 10 monclient: init 2024-03-25T18:09:46.737 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.734+0000 7fd61d61e640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:46.737 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.734+0000 7fd61d61e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:09:46.737 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.734+0000 7fd61d61e640 10 monclient(hunting): picked mon.noname-a con 0x7fd6180a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:09:46.737 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.734+0000 7fd61d61e640 10 monclient(hunting): picked mon.noname-c con 0x7fd61813f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:09:46.737 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.734+0000 7fd61d61e640 10 monclient(hunting): picked mon.noname-b con 0x7fd6181430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:09:46.737 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.734+0000 7fd61d61e640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd618139070 con 0x7fd6180a98f0 2024-03-25T18:09:46.737 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.734+0000 7fd61d61e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd618133710 con 0x7fd6181430c0 2024-03-25T18:09:46.737 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.734+0000 7fd61d61e640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6181338a0 con 0x7fd61813f710 2024-03-25T18:09:46.737 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.734+0000 7fd61d61e640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:46.737 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.734+0000 7fd61d61e640 10 monclient(hunting): authenticate will time out at 7335.793463s 2024-03-25T18:09:46.738 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.737+0000 7fd6177fe640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fd6181430c0 0x7fd618133000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:40342/0 (socket says 172.21.15.66:40342) 2024-03-25T18:09:46.738 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.737+0000 7fd6177fe640 1 -- 172.21.15.66:0/703003681 learned_addr learned my addr 172.21.15.66:0/703003681 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:09:46.742 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.741+0000 7fd5ff7fe640 10 client.?.objecter ms_handle_connect 0x7fd6181430c0 2024-03-25T18:09:46.742 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.741+0000 7fd5ff7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:46.744 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.744+0000 7fd5ff7fe640 10 client.?.objecter ms_handle_connect 0x7fd6180a98f0 2024-03-25T18:09:46.744 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.744+0000 7fd5ff7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:46.748 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.747+0000 7fd5ff7fe640 1 -- 172.21.15.66:0/703003681 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2940386916 0 0) 0x7fd618133710 con 0x7fd6181430c0 2024-03-25T18:09:46.748 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.747+0000 7fd5ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:46.748 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.747+0000 7fd5ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:46.748 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.747+0000 7fd5ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:46.748 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.747+0000 7fd5ff7fe640 10 monclient(hunting): my global_id is 4483 2024-03-25T18:09:46.748 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.747+0000 7fd5ff7fe640 1 -- 172.21.15.66:0/703003681 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd604004eb0 con 0x7fd6181430c0 2024-03-25T18:09:46.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.762+0000 7fd5ff7fe640 10 client.?.objecter ms_handle_connect 0x7fd61813f710 2024-03-25T18:09:46.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.762+0000 7fd5ff7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:46.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.762+0000 7fd5ff7fe640 1 -- 172.21.15.66:0/703003681 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1020971976 0 0) 0x7fd618139070 con 0x7fd6180a98f0 2024-03-25T18:09:46.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.762+0000 7fd5ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:46.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.762+0000 7fd5ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:46.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.762+0000 7fd5ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:46.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.762+0000 7fd5ff7fe640 10 monclient(hunting): my global_id is 4491 2024-03-25T18:09:46.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.762+0000 7fd5ff7fe640 1 -- 172.21.15.66:0/703003681 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd618133710 con 0x7fd6180a98f0 2024-03-25T18:09:46.769 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.768+0000 7fd5ff7fe640 1 -- 172.21.15.66:0/703003681 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 576704223 0 0) 0x7fd604004eb0 con 0x7fd6181430c0 2024-03-25T18:09:46.769 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.768+0000 7fd5ff7fe640 1 -- 172.21.15.66:0/703003681 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd618139070 con 0x7fd6181430c0 2024-03-25T18:09:46.769 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.768+0000 7fd5ff7fe640 1 -- 172.21.15.66:0/703003681 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fd610001c90 con 0x7fd6181430c0 2024-03-25T18:09:46.769 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.768+0000 7fd5ff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:46.769 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.768+0000 7fd5ff7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:46.769 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.768+0000 7fd5ff7fe640 10 monclient(hunting): dump: 2024-03-25T18:09:46.769 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:46.769 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:46.770 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:46.770 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:46.770 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:46.770 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:46.770 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:46.770 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:46.770 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:46.770 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:46.770 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.768+0000 7fd5ff7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:46.776 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.775+0000 7fd5ff7fe640 1 -- 172.21.15.66:0/703003681 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3230198159 0 0) 0x7fd618133710 con 0x7fd6180a98f0 2024-03-25T18:09:46.776 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.775+0000 7fd5ff7fe640 1 -- 172.21.15.66:0/703003681 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd604004eb0 con 0x7fd6180a98f0 2024-03-25T18:09:46.776 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.775+0000 7fd5ff7fe640 1 -- 172.21.15.66:0/703003681 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fd6080018b0 con 0x7fd6180a98f0 2024-03-25T18:09:46.776 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.775+0000 7fd5ff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:46.776 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.775+0000 7fd5ff7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:46.776 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.775+0000 7fd5ff7fe640 10 monclient(hunting): dump: 2024-03-25T18:09:46.776 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:46.776 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:46.776 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:46.776 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:46.776 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:46.777 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:46.777 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:46.777 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:46.777 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:46.777 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:46.783 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.782+0000 7fd5ff7fe640 1 -- 172.21.15.66:0/703003681 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2034435489 0 0) 0x7fd618139070 con 0x7fd6181430c0 2024-03-25T18:09:46.783 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.782+0000 7fd5ff7fe640 1 -- 172.21.15.66:0/703003681 >> v1:172.21.15.66:6790/0 conn(0x7fd61813f710 legacy=0x7fd6181daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:46.783 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.782+0000 7fd5ff7fe640 1 -- 172.21.15.66:0/703003681 >> v1:172.21.15.66:6789/0 conn(0x7fd6180a98f0 legacy=0x7fd6181378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:46.783 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.782+0000 7fd5ff7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:09:46.783 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.782+0000 7fd5ff7fe640 1 monclient: found mon.b 2024-03-25T18:09:46.783 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.782+0000 7fd5ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:46.783 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.782+0000 7fd5ff7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:09:46.783 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.782+0000 7fd5ff7fe640 1 -- 172.21.15.66:0/703003681 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd618133a30 con 0x7fd6181430c0 2024-03-25T18:09:46.783 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.782+0000 7fd5ff7fe640 10 monclient: _finish_auth 0 2024-03-25T18:09:46.783 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.782+0000 7fd5ff7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:09:46.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.782+0000 7fd5ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:46.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.782+0000 7fd61d61e640 5 monclient: authenticate success, global_id 4483 2024-03-25T18:09:46.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.782+0000 7fd61d61e640 10 monclient: _renew_subs 2024-03-25T18:09:46.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.782+0000 7fd61d61e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:09:46.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.782+0000 7fd61d61e640 1 -- 172.21.15.66:0/703003681 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6180b7100 con 0x7fd6181430c0 2024-03-25T18:09:46.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.782+0000 7fd61d61e640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:46.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.782+0000 7fd61d61e640 10 monclient: _renew_subs 2024-03-25T18:09:46.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.782+0000 7fd61d61e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:09:46.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.782+0000 7fd61d61e640 1 -- 172.21.15.66:0/703003681 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd618136e60 con 0x7fd6181430c0 2024-03-25T18:09:46.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.782+0000 7fd61d61e640 1 librados: init done 2024-03-25T18:09:46.785 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.783+0000 7fd61d61e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:46.785 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.783+0000 7fd61d61e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:46.785 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.783+0000 7fd61d61e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:09:46.785 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:46.783+0000 7fd61d61e640 1 -- 172.21.15.66:0/703003681 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6180627b0 con 0x7fd6181430c0 2024-03-25T18:09:47.057 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.056+0000 7fd5ff7fe640 1 -- 172.21.15.66:0/703003681 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd610003990 con 0x7fd6181430c0 2024-03-25T18:09:47.057 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.056+0000 7fd5ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:47.057 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.056+0000 7fd5ff7fe640 1 -- 172.21.15.66:0/703003681 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fd610003ce0 con 0x7fd6181430c0 2024-03-25T18:09:47.057 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.056+0000 7fd5ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:47.057 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.056+0000 7fd5ff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:47.057 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.056+0000 7fd5ff7fe640 10 monclient: dump: 2024-03-25T18:09:47.057 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:47.057 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:47.057 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:47.057 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:47.057 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:47.057 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:47.057 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:47.057 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:47.058 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:47.058 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:47.058 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.056+0000 7fd6157fa640 4 set_mon_vals no callback set 2024-03-25T18:09:47.058 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.057+0000 7fd5ff7fe640 1 -- 172.21.15.66:0/703003681 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7fd610003f00 con 0x7fd6181430c0 2024-03-25T18:09:47.058 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.058+0000 7fd616ffd640 10 monclient: get_auth_request con 0x7fd60406de90 auth_method 0 2024-03-25T18:09:47.058 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.058+0000 7fd5ff7fe640 10 client.4483.objecter ms_handle_connect 0x7fd60406de90 2024-03-25T18:09:47.063 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.063+0000 7fd5ff7fe640 1 -- 172.21.15.66:0/703003681 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7fd610054b20 con 0x7fd6181430c0 2024-03-25T18:09:47.063 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.063+0000 7fd5ff7fe640 10 client.4483.objecter ms_dispatch 0x7fd618000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:47.063 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.063+0000 7fd5ff7fe640 3 client.4483.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:47.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.063+0000 7fd5ff7fe640 3 client.4483.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:47.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.063+0000 7fd5ff7fe640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-25T18:09:47.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.063+0000 7fd5ff7fe640 1 -- 172.21.15.66:0/703003681 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd610050630 con 0x7fd6181430c0 2024-03-25T18:09:47.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.063+0000 7fd5ff7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:47.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.063+0000 7fd5ff7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:47.321 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.320+0000 7fd61d61e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:47.321 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.320+0000 7fd61d61e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:47.321 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.320+0000 7fd61d61e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:09:47.321 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.320+0000 7fd61d61e640 1 -- 172.21.15.66:0/703003681 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd618074cc0 con 0x7fd6181430c0 2024-03-25T18:09:47.345 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.343+0000 7fd5ff7fe640 1 -- 172.21.15.66:0/703003681 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7fd610052560 con 0x7fd6181430c0 2024-03-25T18:09:47.345 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.343+0000 7fd5ff7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:47.345 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.343+0000 7fd5ff7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:47.346 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:09:47.346 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:47.347 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.346+0000 7fd61d61e640 10 librados: watch_flush enter 2024-03-25T18:09:47.347 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.346+0000 7fd61d61e640 10 librados: watch_flush exit 2024-03-25T18:09:47.347 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.346+0000 7fd61d61e640 20 client.4483.objecter shutdown clearing up homeless session... 2024-03-25T18:09:47.347 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.346+0000 7fd61d61e640 10 client.4483.objecter successfully canceled tick 2024-03-25T18:09:47.347 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.346+0000 7fd61d61e640 1 -- 172.21.15.66:0/703003681 >> v1:172.21.15.193:6800/2275676803 conn(0x7fd60406de90 legacy=0x7fd604070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:47.347 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.346+0000 7fd61d61e640 10 monclient: shutdown 2024-03-25T18:09:47.347 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.346+0000 7fd61d61e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:47.347 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.346+0000 7fd61d61e640 1 -- 172.21.15.66:0/703003681 >> v1:172.21.15.193:6789/0 conn(0x7fd6181430c0 legacy=0x7fd618133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:47.347 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.346+0000 7fd61d61e640 1 -- 172.21.15.66:0/703003681 shutdown_connections 2024-03-25T18:09:47.347 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.346+0000 7fd61d61e640 1 -- 172.21.15.66:0/703003681 >> 172.21.15.66:0/703003681 conn(0x7fd6180b7ea0 msgr2=0x7fd618074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:47.347 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.346+0000 7fd61d61e640 1 -- 172.21.15.66:0/703003681 shutdown_connections 2024-03-25T18:09:47.347 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.346+0000 7fd61d61e640 1 -- 172.21.15.66:0/703003681 wait complete. 2024-03-25T18:09:47.348 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:47.346+0000 7fd61d61e640 1 librados: shutdown 2024-03-25T18:09:47.357 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:49.613 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:06:20.593 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:49.623 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:06:20.603 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:49.651 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:06:20.631 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:49.670 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:06:20.629 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:49.676 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:06:20.635 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:49.698 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:06:20.678 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:49.705 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:06:20.685 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:49.705 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:06:20.664 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:49.730 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:06:20.711 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:54.164 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:54.282 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.282+0000 7f7027ff6640 10 monclient: get_monmap_and_config 2024-03-25T18:09:54.282 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.282+0000 7f7027ff6640 10 monclient: build_initial_monmap 2024-03-25T18:09:54.283 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.282+0000 7f7027ff6640 10 monclient: monmap: 2024-03-25T18:09:54.283 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:09:54.283 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:54.283 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:09:54.282669+0000 2024-03-25T18:09:54.283 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:09:54.282669+0000 2024-03-25T18:09:54.283 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:54.283 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:54.283 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:09:54.283 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:09:54.283 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:09:54.283 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:54.284 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.282+0000 7f7027ff6640 1 Processor -- start 2024-03-25T18:09:54.284 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.283+0000 7f7027ff6640 1 -- start start 2024-03-25T18:09:54.285 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.283+0000 7f7027ff6640 10 monclient: init 2024-03-25T18:09:54.285 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.283+0000 7f7027ff6640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:54.285 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.283+0000 7f7027ff6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:09:54.285 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.283+0000 7f7027ff6640 10 monclient(hunting): picked mon.noname-a con 0x7f70200a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:09:54.285 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.283+0000 7f7027ff6640 10 monclient(hunting): picked mon.noname-b con 0x7f702013f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:09:54.285 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.284+0000 7f7027ff6640 10 monclient(hunting): picked mon.noname-c con 0x7f70201430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:09:54.285 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.284+0000 7f7027ff6640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7020142520 con 0x7f70200a98f0 2024-03-25T18:09:54.285 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.284+0000 7f7027ff6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70200b28a0 con 0x7f70201430c0 2024-03-25T18:09:54.285 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.284+0000 7f7027ff6640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70200b2a30 con 0x7f702013f710 2024-03-25T18:09:54.285 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.284+0000 7f7027ff6640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:54.285 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.284+0000 7f7027ff6640 10 monclient(hunting): authenticate will time out at 7343.343175s 2024-03-25T18:09:54.292 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.292+0000 7f7026593640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f70201430c0 0x7f702014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:39968/0 (socket says 172.21.15.66:39968) 2024-03-25T18:09:54.292 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.292+0000 7f7026593640 1 -- 172.21.15.66:0/2960362538 learned_addr learned my addr 172.21.15.66:0/2960362538 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:09:54.305 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.305+0000 7f7024d90640 1 -- 172.21.15.66:0/2960362538 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1446457881 0 0) 0x7f70200b28a0 con 0x7f70201430c0 2024-03-25T18:09:54.305 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.305+0000 7f7024d90640 10 monclient(hunting): _negotiate 2024-03-25T18:09:54.305 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.305+0000 7f7024d90640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:54.305 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.305+0000 7f7024d90640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:54.305 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.305+0000 7f7024d90640 10 monclient(hunting): my global_id is 4486 2024-03-25T18:09:54.305 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.305+0000 7f7024d90640 1 -- 172.21.15.66:0/2960362538 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f700c003520 con 0x7f70201430c0 2024-03-25T18:09:54.315 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.314+0000 7f7024d90640 1 -- 172.21.15.66:0/2960362538 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931601952 0 0) 0x7f70200b2a30 con 0x7f702013f710 2024-03-25T18:09:54.315 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.314+0000 7f7024d90640 10 monclient(hunting): _negotiate 2024-03-25T18:09:54.315 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.314+0000 7f7024d90640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:54.315 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.314+0000 7f7024d90640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:54.315 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.314+0000 7f7024d90640 10 monclient(hunting): my global_id is 4487 2024-03-25T18:09:54.315 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.314+0000 7f7024d90640 1 -- 172.21.15.66:0/2960362538 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70200b28a0 con 0x7f702013f710 2024-03-25T18:09:54.324 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.323+0000 7f7024d90640 1 -- 172.21.15.66:0/2960362538 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 544053060 0 0) 0x7f7020142520 con 0x7f70200a98f0 2024-03-25T18:09:54.324 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.323+0000 7f7024d90640 10 monclient(hunting): _negotiate 2024-03-25T18:09:54.324 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.323+0000 7f7024d90640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:54.324 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.323+0000 7f7024d90640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:54.324 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.323+0000 7f7024d90640 10 monclient(hunting): my global_id is 4494 2024-03-25T18:09:54.324 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.323+0000 7f7024d90640 1 -- 172.21.15.66:0/2960362538 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70200b2a30 con 0x7f70200a98f0 2024-03-25T18:09:54.324 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.323+0000 7f7024d90640 1 -- 172.21.15.66:0/2960362538 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1641544637 0 0) 0x7f700c003520 con 0x7f70201430c0 2024-03-25T18:09:54.324 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.323+0000 7f7024d90640 1 -- 172.21.15.66:0/2960362538 >> v1:172.21.15.66:6790/0 conn(0x7f702013f710 legacy=0x7f70201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:54.325 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.323+0000 7f7024d90640 1 -- 172.21.15.66:0/2960362538 >> v1:172.21.15.66:6789/0 conn(0x7f70200a98f0 legacy=0x7f70200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:54.325 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.323+0000 7f7024d90640 10 monclient: _finish_hunting 0 2024-03-25T18:09:54.325 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.323+0000 7f7024d90640 1 monclient: found mon.noname-c 2024-03-25T18:09:54.325 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.323+0000 7f7024d90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:54.325 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.323+0000 7f7024d90640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:09:54.325 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.323+0000 7f7024d90640 1 -- 172.21.15.66:0/2960362538 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70200b44c0 con 0x7f70201430c0 2024-03-25T18:09:54.325 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.323+0000 7f7024d90640 10 monclient: _finish_auth 0 2024-03-25T18:09:54.325 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.323+0000 7f7024d90640 10 monclient: _check_auth_tickets 2024-03-25T18:09:54.325 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.323+0000 7f7024d90640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:54.325 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.323+0000 7f7024d90640 1 -- 172.21.15.66:0/2960362538 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7018001a10 con 0x7f70201430c0 2024-03-25T18:09:54.325 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.323+0000 7f7024d90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:54.325 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.323+0000 7f7024d90640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:54.325 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.323+0000 7f7024d90640 10 monclient: dump: 2024-03-25T18:09:54.325 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:54.326 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:54.326 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:54.326 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:54.326 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:54.326 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:54.326 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:54.326 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:54.326 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:54.326 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:54.326 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.323+0000 7f7027ff6640 5 monclient: authenticate success, global_id 4486 2024-03-25T18:09:54.326 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.323+0000 7f7027ff6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:54.333 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.333+0000 7f7024d90640 1 -- 172.21.15.66:0/2960362538 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7018001bf0 con 0x7f70201430c0 2024-03-25T18:09:54.334 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.333+0000 7f7024d90640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:54.334 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.333+0000 7f7024d90640 1 -- 172.21.15.66:0/2960362538 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7018002000 con 0x7f70201430c0 2024-03-25T18:09:54.334 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.333+0000 7f7027ff6640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:54.334 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.333+0000 7f7027ff6640 4 set_mon_vals no callback set 2024-03-25T18:09:54.334 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.333+0000 7f7024d90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:54.334 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.333+0000 7f7024d90640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:54.334 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.333+0000 7f7024d90640 10 monclient: dump: 2024-03-25T18:09:54.334 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:54.334 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:54.334 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:54.334 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:54.334 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:54.334 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:54.335 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:54.335 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:54.335 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:54.335 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:54.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.334+0000 7f7027ff6640 10 monclient: shutdown 2024-03-25T18:09:54.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.334+0000 7f7027ff6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:54.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.334+0000 7f7027ff6640 1 -- 172.21.15.66:0/2960362538 >> v1:172.21.15.193:6789/0 conn(0x7f70201430c0 legacy=0x7f702014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:54.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.334+0000 7f7027ff6640 1 -- 172.21.15.66:0/2960362538 shutdown_connections 2024-03-25T18:09:54.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.334+0000 7f7027ff6640 1 -- 172.21.15.66:0/2960362538 >> 172.21.15.66:0/2960362538 conn(0x7f70200b7ea0 msgr2=0x7f70200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:54.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.334+0000 7f7027ff6640 1 -- 172.21.15.66:0/2960362538 shutdown_connections 2024-03-25T18:09:54.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.334+0000 7f7027ff6640 1 -- 172.21.15.66:0/2960362538 wait complete. 2024-03-25T18:09:54.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.335+0000 7f7027ff6640 10 monclient: build_initial_monmap 2024-03-25T18:09:54.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.335+0000 7f7027ff6640 10 monclient: monmap: 2024-03-25T18:09:54.337 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:09:54.338 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:54.338 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:09:54.338 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:09:54.338 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:54.338 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:09:54.338 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:09:54.338 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:09:54.338 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:09:54.338 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:54.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.335+0000 7f7027ff6640 1 librados: starting msgr at 2024-03-25T18:09:54.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.335+0000 7f7027ff6640 1 librados: starting objecter 2024-03-25T18:09:54.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.335+0000 7f7027ff6640 1 Processor -- start 2024-03-25T18:09:54.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.335+0000 7f7027ff6640 1 -- start start 2024-03-25T18:09:54.338 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.335+0000 7f7027ff6640 1 librados: setting wanted keys 2024-03-25T18:09:54.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.335+0000 7f7027ff6640 1 librados: calling monclient init 2024-03-25T18:09:54.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.335+0000 7f7027ff6640 10 monclient: init 2024-03-25T18:09:54.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.336+0000 7f7027ff6640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:54.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.336+0000 7f7027ff6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:09:54.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.336+0000 7f7027ff6640 10 monclient(hunting): picked mon.noname-b con 0x7f70200a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:09:54.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.336+0000 7f7027ff6640 10 monclient(hunting): picked mon.noname-c con 0x7f702013f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:09:54.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.336+0000 7f7027ff6640 10 monclient(hunting): picked mon.noname-a con 0x7f70201430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:09:54.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.336+0000 7f7027ff6640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f702013c2b0 con 0x7f70201430c0 2024-03-25T18:09:54.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.336+0000 7f7027ff6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7020136f10 con 0x7f70200a98f0 2024-03-25T18:09:54.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.336+0000 7f7027ff6640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70201370a0 con 0x7f702013f710 2024-03-25T18:09:54.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.336+0000 7f7027ff6640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:54.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.336+0000 7f7027ff6640 10 monclient(hunting): authenticate will time out at 7343.395813s 2024-03-25T18:09:54.344 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.344+0000 7f7025d92640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f70200a98f0 0x7f70201d7590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:40010/0 (socket says 172.21.15.66:40010) 2024-03-25T18:09:54.344 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.344+0000 7f7025d92640 1 -- 172.21.15.66:0/814684791 learned_addr learned my addr 172.21.15.66:0/814684791 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:09:54.345 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.345+0000 7f70167fc640 10 client.?.objecter ms_handle_connect 0x7f70200a98f0 2024-03-25T18:09:54.345 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.345+0000 7f70167fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:54.354 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.353+0000 7f70167fc640 10 client.?.objecter ms_handle_connect 0x7f702013f710 2024-03-25T18:09:54.354 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.353+0000 7f70167fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:54.360 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.359+0000 7f70167fc640 1 -- 172.21.15.66:0/814684791 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 954972749 0 0) 0x7f70201370a0 con 0x7f702013f710 2024-03-25T18:09:54.360 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.359+0000 7f70167fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:54.360 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.359+0000 7f70167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:54.360 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.359+0000 7f70167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:54.360 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.359+0000 7f70167fc640 10 monclient(hunting): my global_id is 4490 2024-03-25T18:09:54.360 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.359+0000 7f70167fc640 1 -- 172.21.15.66:0/814684791 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7004004eb0 con 0x7f702013f710 2024-03-25T18:09:54.361 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.361+0000 7f70167fc640 1 -- 172.21.15.66:0/814684791 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2443711257 0 0) 0x7f7020136f10 con 0x7f70200a98f0 2024-03-25T18:09:54.362 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.361+0000 7f70167fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:54.362 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.361+0000 7f70167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:54.362 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.361+0000 7f70167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:54.362 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.361+0000 7f70167fc640 10 monclient(hunting): my global_id is 4489 2024-03-25T18:09:54.362 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.361+0000 7f70167fc640 1 -- 172.21.15.66:0/814684791 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70201370a0 con 0x7f70200a98f0 2024-03-25T18:09:54.363 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.362+0000 7f70167fc640 10 client.?.objecter ms_handle_connect 0x7f70201430c0 2024-03-25T18:09:54.363 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.362+0000 7f70167fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:54.372 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.371+0000 7f70167fc640 1 -- 172.21.15.66:0/814684791 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3759523388 0 0) 0x7f702013c2b0 con 0x7f70201430c0 2024-03-25T18:09:54.372 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.371+0000 7f70167fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:54.372 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.371+0000 7f70167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:54.372 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.371+0000 7f70167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:54.372 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.371+0000 7f70167fc640 10 monclient(hunting): my global_id is 4497 2024-03-25T18:09:54.372 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.371+0000 7f70167fc640 1 -- 172.21.15.66:0/814684791 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7020136f10 con 0x7f70201430c0 2024-03-25T18:09:54.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.378+0000 7f70167fc640 1 -- 172.21.15.66:0/814684791 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 449375042 0 0) 0x7f70201370a0 con 0x7f70200a98f0 2024-03-25T18:09:54.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.378+0000 7f70167fc640 1 -- 172.21.15.66:0/814684791 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f702013c2b0 con 0x7f70200a98f0 2024-03-25T18:09:54.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.378+0000 7f70167fc640 1 -- 172.21.15.66:0/814684791 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f701c0019f0 con 0x7f70200a98f0 2024-03-25T18:09:54.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.378+0000 7f70167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:54.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.378+0000 7f70167fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:54.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.378+0000 7f70167fc640 10 monclient(hunting): dump: 2024-03-25T18:09:54.380 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:54.380 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:54.380 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:54.380 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:54.380 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:54.380 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:54.380 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:54.380 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:54.380 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:54.380 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:54.380 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.379+0000 7f70167fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:54.381 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.380+0000 7f70167fc640 1 -- 172.21.15.66:0/814684791 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4024078681 0 0) 0x7f7004004eb0 con 0x7f702013f710 2024-03-25T18:09:54.381 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.380+0000 7f70167fc640 1 -- 172.21.15.66:0/814684791 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70201370a0 con 0x7f702013f710 2024-03-25T18:09:54.381 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.380+0000 7f70167fc640 1 -- 172.21.15.66:0/814684791 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f70100018b0 con 0x7f702013f710 2024-03-25T18:09:54.381 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.380+0000 7f70167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:54.381 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.380+0000 7f70167fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:54.381 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.380+0000 7f70167fc640 10 monclient(hunting): dump: 2024-03-25T18:09:54.381 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:54.381 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:54.381 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:54.381 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:54.382 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:54.382 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:54.382 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:54.382 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:54.382 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:54.382 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:54.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.394+0000 7f70167fc640 1 -- 172.21.15.66:0/814684791 <== mon.2 v1:172.21.15.66:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1949181697 0 0) 0x7f70201370a0 con 0x7f702013f710 2024-03-25T18:09:54.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.394+0000 7f70167fc640 1 -- 172.21.15.66:0/814684791 >> v1:172.21.15.193:6789/0 conn(0x7f70200a98f0 legacy=0x7f70201d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:54.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.394+0000 7f70167fc640 1 -- 172.21.15.66:0/814684791 >> v1:172.21.15.66:6789/0 conn(0x7f70201430c0 legacy=0x7f7020133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:54.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.394+0000 7f70167fc640 10 monclient: _finish_hunting 0 2024-03-25T18:09:54.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.394+0000 7f70167fc640 1 monclient: found mon.c 2024-03-25T18:09:54.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.394+0000 7f70167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:54.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.394+0000 7f70167fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:09:54.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.394+0000 7f70167fc640 1 -- 172.21.15.66:0/814684791 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7020137230 con 0x7f702013f710 2024-03-25T18:09:54.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.394+0000 7f70167fc640 10 monclient: _finish_auth 0 2024-03-25T18:09:54.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.394+0000 7f70167fc640 10 monclient: _check_auth_tickets 2024-03-25T18:09:54.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.394+0000 7f70167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:54.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.394+0000 7f7027ff6640 5 monclient: authenticate success, global_id 4490 2024-03-25T18:09:54.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.394+0000 7f7027ff6640 10 monclient: _renew_subs 2024-03-25T18:09:54.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.394+0000 7f7027ff6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:09:54.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.394+0000 7f7027ff6640 1 -- 172.21.15.66:0/814684791 --> v1:172.21.15.66:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f702013b620 con 0x7f702013f710 2024-03-25T18:09:54.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.394+0000 7f7027ff6640 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:54.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.394+0000 7f7027ff6640 10 monclient: _renew_subs 2024-03-25T18:09:54.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.394+0000 7f7027ff6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:09:54.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.394+0000 7f7027ff6640 1 -- 172.21.15.66:0/814684791 --> v1:172.21.15.66:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7020133710 con 0x7f702013f710 2024-03-25T18:09:54.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.394+0000 7f7027ff6640 1 librados: init done 2024-03-25T18:09:54.397 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.395+0000 7f7027ff6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:54.397 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.395+0000 7f7027ff6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:54.397 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.395+0000 7f7027ff6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:09:54.397 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.395+0000 7f7027ff6640 1 -- 172.21.15.66:0/814684791 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70200627b0 con 0x7f702013f710 2024-03-25T18:09:54.697 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.696+0000 7f70167fc640 1 -- 172.21.15.66:0/814684791 <== mon.2 v1:172.21.15.66:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70100035f0 con 0x7f702013f710 2024-03-25T18:09:54.697 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.696+0000 7f70167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:54.697 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.696+0000 7f70167fc640 1 -- 172.21.15.66:0/814684791 <== mon.2 v1:172.21.15.66:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7010003760 con 0x7f702013f710 2024-03-25T18:09:54.697 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.696+0000 7f70167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:54.697 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.696+0000 7f70167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:54.697 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.696+0000 7f70167fc640 10 monclient: dump: 2024-03-25T18:09:54.698 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:09:54.698 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:09:54.698 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:54.698 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:09:54.698 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:54.698 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:09:54.698 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:09:54.698 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:09:54.698 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:09:54.698 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:09:54.698 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.696+0000 7f7017fff640 4 set_mon_vals no callback set 2024-03-25T18:09:54.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.697+0000 7f70167fc640 1 -- 172.21.15.66:0/814684791 <== mon.2 v1:172.21.15.66:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f7010019730 con 0x7f702013f710 2024-03-25T18:09:54.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.698+0000 7f7025d92640 10 monclient: get_auth_request con 0x7f700406df20 auth_method 0 2024-03-25T18:09:54.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.698+0000 7f70167fc640 10 client.4490.objecter ms_handle_connect 0x7f700406df20 2024-03-25T18:09:54.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.703+0000 7f70167fc640 1 -- 172.21.15.66:0/814684791 <== mon.2 v1:172.21.15.66:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f7010053490 con 0x7f702013f710 2024-03-25T18:09:54.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.703+0000 7f70167fc640 10 client.4490.objecter ms_dispatch 0x7f7020000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:54.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.703+0000 7f70167fc640 3 client.4490.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:54.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.703+0000 7f70167fc640 3 client.4490.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:54.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.703+0000 7f70167fc640 20 client.4490.objecter dump_active .. 0 homeless 2024-03-25T18:09:54.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.703+0000 7f70167fc640 1 -- 172.21.15.66:0/814684791 <== mon.2 v1:172.21.15.66:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f701004edd0 con 0x7f702013f710 2024-03-25T18:09:54.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.703+0000 7f70167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:54.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.703+0000 7f70167fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:54.974 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.973+0000 7f7027ff6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:54.974 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.973+0000 7f7027ff6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:54.974 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.973+0000 7f7027ff6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:09:54.974 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:54.973+0000 7f7027ff6640 1 -- 172.21.15.66:0/814684791 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f702014f240 con 0x7f702013f710 2024-03-25T18:09:55.012 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:55.011+0000 7f70167fc640 1 -- 172.21.15.66:0/814684791 <== mon.2 v1:172.21.15.66:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f7010050ed0 con 0x7f702013f710 2024-03-25T18:09:55.012 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:55.012+0000 7f70167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:55.012 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:55.012+0000 7f70167fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:55.012 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:09:55.012 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:55.015 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:55.014+0000 7f7027ff6640 10 librados: watch_flush enter 2024-03-25T18:09:55.015 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:55.014+0000 7f7027ff6640 10 librados: watch_flush exit 2024-03-25T18:09:55.015 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:55.014+0000 7f7027ff6640 20 client.4490.objecter shutdown clearing up homeless session... 2024-03-25T18:09:55.015 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:55.014+0000 7f7027ff6640 10 client.4490.objecter successfully canceled tick 2024-03-25T18:09:55.015 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:55.014+0000 7f7027ff6640 1 -- 172.21.15.66:0/814684791 >> v1:172.21.15.193:6800/2275676803 conn(0x7f700406df20 legacy=0x7f70040703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:55.016 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:55.014+0000 7f7027ff6640 10 monclient: shutdown 2024-03-25T18:09:55.016 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:55.014+0000 7f7027ff6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:55.016 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:55.014+0000 7f7027ff6640 1 -- 172.21.15.66:0/814684791 >> v1:172.21.15.66:6790/0 conn(0x7f702013f710 legacy=0x7f70201377d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:55.016 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:55.014+0000 7f7027ff6640 1 -- 172.21.15.66:0/814684791 shutdown_connections 2024-03-25T18:09:55.016 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:55.014+0000 7f7027ff6640 1 -- 172.21.15.66:0/814684791 >> 172.21.15.66:0/814684791 conn(0x7f70200b7ea0 msgr2=0x7f7020074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:55.016 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:55.014+0000 7f7027ff6640 1 -- 172.21.15.66:0/814684791 shutdown_connections 2024-03-25T18:09:55.016 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:55.014+0000 7f7027ff6640 1 -- 172.21.15.66:0/814684791 wait complete. 2024-03-25T18:09:55.016 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:09:55.014+0000 7f7027ff6640 1 librados: shutdown 2024-03-25T18:09:55.027 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:01.834 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:01.953 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.952+0000 7f03b980f640 10 monclient: get_monmap_and_config 2024-03-25T18:10:01.953 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.952+0000 7f03b980f640 10 monclient: build_initial_monmap 2024-03-25T18:10:01.953 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.952+0000 7f03b980f640 10 monclient: monmap: 2024-03-25T18:10:01.954 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:10:01.954 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:01.954 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:10:01.953452+0000 2024-03-25T18:10:01.954 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:10:01.953452+0000 2024-03-25T18:10:01.954 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:01.954 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:01.954 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:10:01.954 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:10:01.954 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:10:01.954 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:01.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.953+0000 7f03b980f640 1 Processor -- start 2024-03-25T18:10:01.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.953+0000 7f03b980f640 1 -- start start 2024-03-25T18:10:01.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.953+0000 7f03b980f640 10 monclient: init 2024-03-25T18:10:01.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.954+0000 7f03b980f640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:01.956 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.954+0000 7f03b980f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:10:01.956 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.954+0000 7f03b980f640 10 monclient(hunting): picked mon.noname-c con 0x7f03b40a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:10:01.956 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.954+0000 7f03b980f640 10 monclient(hunting): picked mon.noname-a con 0x7f03b413f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:10:01.956 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.954+0000 7f03b980f640 10 monclient(hunting): picked mon.noname-b con 0x7f03b41430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:10:01.956 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.954+0000 7f03b980f640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03b4142520 con 0x7f03b413f710 2024-03-25T18:10:01.956 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.954+0000 7f03b980f640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03b40b28a0 con 0x7f03b40a98f0 2024-03-25T18:10:01.956 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.954+0000 7f03b980f640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03b40b2a30 con 0x7f03b41430c0 2024-03-25T18:10:01.956 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.954+0000 7f03b980f640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:01.956 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.954+0000 7f03b980f640 10 monclient(hunting): authenticate will time out at 7351.013878s 2024-03-25T18:10:01.966 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.966+0000 7f03b27fc640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f03b413f710 0x7f03b41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:39510/0 (socket says 172.21.15.66:39510) 2024-03-25T18:10:01.966 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.966+0000 7f03b27fc640 1 -- 172.21.15.66:0/644151673 learned_addr learned my addr 172.21.15.66:0/644151673 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:10:01.990 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.990+0000 7f03b1ffb640 1 -- 172.21.15.66:0/644151673 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2869967682 0 0) 0x7f03b4142520 con 0x7f03b413f710 2024-03-25T18:10:01.990 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.990+0000 7f03b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:01.990 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.990+0000 7f03b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:01.990 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.990+0000 7f03b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:01.991 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.990+0000 7f03b1ffb640 10 monclient(hunting): my global_id is 4500 2024-03-25T18:10:01.991 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.990+0000 7f03b1ffb640 1 -- 172.21.15.66:0/644151673 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f039c003520 con 0x7f03b413f710 2024-03-25T18:10:01.992 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.991+0000 7f03b1ffb640 1 -- 172.21.15.66:0/644151673 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2443950615 0 0) 0x7f03b40b28a0 con 0x7f03b40a98f0 2024-03-25T18:10:01.992 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.991+0000 7f03b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:01.992 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.991+0000 7f03b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:01.992 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.991+0000 7f03b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:01.992 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.991+0000 7f03b1ffb640 10 monclient(hunting): my global_id is 4492 2024-03-25T18:10:01.992 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.991+0000 7f03b1ffb640 1 -- 172.21.15.66:0/644151673 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03b4142520 con 0x7f03b40a98f0 2024-03-25T18:10:01.995 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.994+0000 7f03b1ffb640 1 -- 172.21.15.66:0/644151673 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3315048988 0 0) 0x7f03b40b2a30 con 0x7f03b41430c0 2024-03-25T18:10:01.995 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.994+0000 7f03b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:01.995 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.994+0000 7f03b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:01.995 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.994+0000 7f03b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:01.995 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.994+0000 7f03b1ffb640 10 monclient(hunting): my global_id is 4493 2024-03-25T18:10:01.995 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.994+0000 7f03b1ffb640 1 -- 172.21.15.66:0/644151673 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03b40b28a0 con 0x7f03b41430c0 2024-03-25T18:10:02.000 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.999+0000 7f03b1ffb640 1 -- 172.21.15.66:0/644151673 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1715018839 0 0) 0x7f03b4142520 con 0x7f03b40a98f0 2024-03-25T18:10:02.000 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.999+0000 7f03b1ffb640 1 -- 172.21.15.66:0/644151673 >> v1:172.21.15.66:6790/0 conn(0x7f03b41430c0 legacy=0x7f03b414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:02.000 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.999+0000 7f03b1ffb640 1 -- 172.21.15.66:0/644151673 >> v1:172.21.15.66:6789/0 conn(0x7f03b413f710 legacy=0x7f03b41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:02.000 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.999+0000 7f03b1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:10:02.000 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.999+0000 7f03b1ffb640 1 monclient: found mon.noname-c 2024-03-25T18:10:02.000 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.999+0000 7f03b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:02.000 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.999+0000 7f03b1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:10:02.000 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.999+0000 7f03b1ffb640 1 -- 172.21.15.66:0/644151673 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b40b44c0 con 0x7f03b40a98f0 2024-03-25T18:10:02.000 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.999+0000 7f03b1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:10:02.000 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.999+0000 7f03b1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:10:02.000 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.999+0000 7f03b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:02.000 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.999+0000 7f03b1ffb640 1 -- 172.21.15.66:0/644151673 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f03a4001670 con 0x7f03b40a98f0 2024-03-25T18:10:02.000 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.999+0000 7f03b980f640 5 monclient: authenticate success, global_id 4492 2024-03-25T18:10:02.001 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.999+0000 7f03b980f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:02.001 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.999+0000 7f03b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:02.001 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.999+0000 7f03b1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:02.001 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.999+0000 7f03b1ffb640 10 monclient: dump: 2024-03-25T18:10:02.001 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:02.001 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:02.001 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:02.001 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:02.001 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:02.001 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:02.001 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:02.001 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:02.001 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:02.001 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:02.001 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:01.999+0000 7f03b980f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:02.009 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.009+0000 7f03b1ffb640 1 -- 172.21.15.66:0/644151673 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03a40017d0 con 0x7f03b40a98f0 2024-03-25T18:10:02.009 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.009+0000 7f03b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:02.009 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.009+0000 7f03b1ffb640 1 -- 172.21.15.66:0/644151673 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f03a4001be0 con 0x7f03b40a98f0 2024-03-25T18:10:02.009 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.009+0000 7f03b980f640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:02.010 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.009+0000 7f03b980f640 4 set_mon_vals no callback set 2024-03-25T18:10:02.010 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.009+0000 7f03b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:02.010 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.009+0000 7f03b1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:02.010 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.009+0000 7f03b1ffb640 10 monclient: dump: 2024-03-25T18:10:02.010 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:02.010 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:02.010 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:02.010 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:02.010 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:02.010 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:02.010 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:02.010 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:02.010 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:02.010 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:02.012 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.009+0000 7f03b980f640 10 monclient: shutdown 2024-03-25T18:10:02.012 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.009+0000 7f03b980f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:02.012 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.009+0000 7f03b980f640 1 -- 172.21.15.66:0/644151673 >> v1:172.21.15.193:6789/0 conn(0x7f03b40a98f0 legacy=0x7f03b40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:02.012 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.010+0000 7f03b980f640 1 -- 172.21.15.66:0/644151673 shutdown_connections 2024-03-25T18:10:02.012 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.010+0000 7f03b980f640 1 -- 172.21.15.66:0/644151673 >> 172.21.15.66:0/644151673 conn(0x7f03b40b7ea0 msgr2=0x7f03b40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:02.012 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.010+0000 7f03b980f640 1 -- 172.21.15.66:0/644151673 shutdown_connections 2024-03-25T18:10:02.012 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.010+0000 7f03b980f640 1 -- 172.21.15.66:0/644151673 wait complete. 2024-03-25T18:10:02.012 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.011+0000 7f03b980f640 10 monclient: build_initial_monmap 2024-03-25T18:10:02.012 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.011+0000 7f03b980f640 10 monclient: monmap: 2024-03-25T18:10:02.013 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:10:02.013 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:02.013 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:10:02.013 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:10:02.013 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:02.013 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:10:02.013 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:10:02.013 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:10:02.013 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:10:02.013 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:02.013 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.011+0000 7f03b980f640 1 librados: starting msgr at 2024-03-25T18:10:02.013 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.011+0000 7f03b980f640 1 librados: starting objecter 2024-03-25T18:10:02.013 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.011+0000 7f03b980f640 1 Processor -- start 2024-03-25T18:10:02.013 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.012+0000 7f03b980f640 1 -- start start 2024-03-25T18:10:02.013 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.012+0000 7f03b980f640 1 librados: setting wanted keys 2024-03-25T18:10:02.013 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.012+0000 7f03b980f640 1 librados: calling monclient init 2024-03-25T18:10:02.014 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.012+0000 7f03b980f640 10 monclient: init 2024-03-25T18:10:02.014 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.012+0000 7f03b980f640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:02.015 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.012+0000 7f03b980f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:10:02.015 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.012+0000 7f03b980f640 10 monclient(hunting): picked mon.noname-c con 0x7f03b40a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:10:02.015 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.012+0000 7f03b980f640 10 monclient(hunting): picked mon.noname-a con 0x7f03b413f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:10:02.015 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.012+0000 7f03b980f640 10 monclient(hunting): picked mon.noname-b con 0x7f03b41430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:10:02.015 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.012+0000 7f03b980f640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03b4139330 con 0x7f03b413f710 2024-03-25T18:10:02.015 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.012+0000 7f03b980f640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03b40abe70 con 0x7f03b41430c0 2024-03-25T18:10:02.015 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.012+0000 7f03b980f640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03b40ac150 con 0x7f03b40a98f0 2024-03-25T18:10:02.015 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.012+0000 7f03b980f640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:02.015 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.013+0000 7f03b980f640 10 monclient(hunting): authenticate will time out at 7351.072090s 2024-03-25T18:10:02.019 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.018+0000 7f03b37fe640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f03b41430c0 0x7f03b41e2510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:44704/0 (socket says 172.21.15.66:44704) 2024-03-25T18:10:02.019 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.019+0000 7f03b37fe640 1 -- 172.21.15.66:0/2331211309 learned_addr learned my addr 172.21.15.66:0/2331211309 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:10:02.019 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.019+0000 7f039bfff640 10 client.?.objecter ms_handle_connect 0x7f03b41430c0 2024-03-25T18:10:02.019 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.019+0000 7f039bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:02.022 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.021+0000 7f039bfff640 10 client.?.objecter ms_handle_connect 0x7f03b40a98f0 2024-03-25T18:10:02.022 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.021+0000 7f039bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:02.029 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.028+0000 7f039bfff640 10 client.?.objecter ms_handle_connect 0x7f03b413f710 2024-03-25T18:10:02.029 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.028+0000 7f039bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:02.031 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.030+0000 7f039bfff640 1 -- 172.21.15.66:0/2331211309 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754793218 0 0) 0x7f03b40abe70 con 0x7f03b41430c0 2024-03-25T18:10:02.031 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.030+0000 7f039bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:02.031 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.030+0000 7f039bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:02.031 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.030+0000 7f039bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:02.031 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.030+0000 7f039bfff640 10 monclient(hunting): my global_id is 4495 2024-03-25T18:10:02.031 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.030+0000 7f039bfff640 1 -- 172.21.15.66:0/2331211309 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03a0004eb0 con 0x7f03b41430c0 2024-03-25T18:10:02.040 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.039+0000 7f039bfff640 1 -- 172.21.15.66:0/2331211309 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 377880257 0 0) 0x7f03b40ac150 con 0x7f03b40a98f0 2024-03-25T18:10:02.040 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.039+0000 7f039bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:02.040 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.039+0000 7f039bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:02.040 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.039+0000 7f039bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:02.040 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.039+0000 7f039bfff640 10 monclient(hunting): my global_id is 4496 2024-03-25T18:10:02.040 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.040+0000 7f039bfff640 1 -- 172.21.15.66:0/2331211309 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03b40abe70 con 0x7f03b40a98f0 2024-03-25T18:10:02.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.051+0000 7f039bfff640 1 -- 172.21.15.66:0/2331211309 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230216052 0 0) 0x7f03b4139330 con 0x7f03b413f710 2024-03-25T18:10:02.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.051+0000 7f039bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:02.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.051+0000 7f039bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:02.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.051+0000 7f039bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:02.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.051+0000 7f039bfff640 10 monclient(hunting): my global_id is 4503 2024-03-25T18:10:02.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.051+0000 7f039bfff640 1 -- 172.21.15.66:0/2331211309 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03b40ac150 con 0x7f03b413f710 2024-03-25T18:10:02.054 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.054+0000 7f039bfff640 1 -- 172.21.15.66:0/2331211309 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4232064762 0 0) 0x7f03a0004eb0 con 0x7f03b41430c0 2024-03-25T18:10:02.054 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.054+0000 7f039bfff640 1 -- 172.21.15.66:0/2331211309 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03b4139330 con 0x7f03b41430c0 2024-03-25T18:10:02.054 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.054+0000 7f039bfff640 1 -- 172.21.15.66:0/2331211309 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f03ac001c70 con 0x7f03b41430c0 2024-03-25T18:10:02.055 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.054+0000 7f039bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:02.055 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.054+0000 7f039bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:02.055 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.054+0000 7f039bfff640 10 monclient(hunting): dump: 2024-03-25T18:10:02.055 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:02.055 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:02.055 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:02.055 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:02.055 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:02.055 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:02.055 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:02.055 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:02.055 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:02.055 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:02.055 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.054+0000 7f039bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:02.066 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.066+0000 7f039bfff640 1 -- 172.21.15.66:0/2331211309 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 52682516 0 0) 0x7f03b40abe70 con 0x7f03b40a98f0 2024-03-25T18:10:02.066 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.066+0000 7f039bfff640 1 -- 172.21.15.66:0/2331211309 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03a0004eb0 con 0x7f03b40a98f0 2024-03-25T18:10:02.066 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.066+0000 7f039bfff640 1 -- 172.21.15.66:0/2331211309 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f03a4001ae0 con 0x7f03b40a98f0 2024-03-25T18:10:02.066 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.066+0000 7f039bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:02.066 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.066+0000 7f039bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:02.066 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.066+0000 7f039bfff640 10 monclient(hunting): dump: 2024-03-25T18:10:02.067 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:02.067 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:02.067 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:02.067 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:02.067 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:02.067 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:02.067 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:02.067 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:02.067 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:02.067 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:02.071 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.070+0000 7f039bfff640 1 -- 172.21.15.66:0/2331211309 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 676752874 0 0) 0x7f03b4139330 con 0x7f03b41430c0 2024-03-25T18:10:02.071 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.070+0000 7f039bfff640 1 -- 172.21.15.66:0/2331211309 >> v1:172.21.15.66:6790/0 conn(0x7f03b40a98f0 legacy=0x7f03b41389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:02.071 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.070+0000 7f039bfff640 1 -- 172.21.15.66:0/2331211309 >> v1:172.21.15.66:6789/0 conn(0x7f03b413f710 legacy=0x7f03b40ac570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:02.071 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.070+0000 7f039bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:10:02.071 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.070+0000 7f039bfff640 1 monclient: found mon.b 2024-03-25T18:10:02.071 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.070+0000 7f039bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:02.071 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.070+0000 7f039bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:10:02.071 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.070+0000 7f039bfff640 1 -- 172.21.15.66:0/2331211309 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b40acc80 con 0x7f03b41430c0 2024-03-25T18:10:02.071 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.070+0000 7f039bfff640 10 monclient: _finish_auth 0 2024-03-25T18:10:02.071 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.070+0000 7f039bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:10:02.071 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.070+0000 7f039bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:02.071 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.070+0000 7f03b980f640 5 monclient: authenticate success, global_id 4495 2024-03-25T18:10:02.071 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.070+0000 7f03b980f640 10 monclient: _renew_subs 2024-03-25T18:10:02.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.070+0000 7f03b980f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:10:02.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.070+0000 7f03b980f640 1 -- 172.21.15.66:0/2331211309 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03b4137e30 con 0x7f03b41430c0 2024-03-25T18:10:02.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.070+0000 7f03b980f640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:02.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.070+0000 7f03b980f640 10 monclient: _renew_subs 2024-03-25T18:10:02.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.070+0000 7f03b980f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:10:02.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.070+0000 7f03b980f640 1 -- 172.21.15.66:0/2331211309 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f03b4136880 con 0x7f03b41430c0 2024-03-25T18:10:02.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.070+0000 7f03b980f640 1 librados: init done 2024-03-25T18:10:02.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.071+0000 7f03b980f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:02.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.071+0000 7f03b980f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:02.073 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.071+0000 7f03b980f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:10:02.073 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.071+0000 7f03b980f640 1 -- 172.21.15.66:0/2331211309 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03b4136b30 con 0x7f03b41430c0 2024-03-25T18:10:02.295 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.294+0000 7f039bfff640 1 -- 172.21.15.66:0/2331211309 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03ac003990 con 0x7f03b41430c0 2024-03-25T18:10:02.295 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.294+0000 7f039bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:02.295 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.294+0000 7f039bfff640 1 -- 172.21.15.66:0/2331211309 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f03ac003da0 con 0x7f03b41430c0 2024-03-25T18:10:02.295 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.294+0000 7f039bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:02.295 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.294+0000 7f039bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:02.295 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.294+0000 7f039bfff640 10 monclient: dump: 2024-03-25T18:10:02.295 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:02.295 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:02.295 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:02.295 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:02.295 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:02.295 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:02.296 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:02.296 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:02.296 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:02.296 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:02.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.294+0000 7f03b17fa640 4 set_mon_vals no callback set 2024-03-25T18:10:02.346 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.345+0000 7f039bfff640 1 -- 172.21.15.66:0/2331211309 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f03ac003f00 con 0x7f03b41430c0 2024-03-25T18:10:02.346 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.346+0000 7f03b2ffd640 10 monclient: get_auth_request con 0x7f03a006df70 auth_method 0 2024-03-25T18:10:02.346 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.346+0000 7f039bfff640 10 client.4495.objecter ms_handle_connect 0x7f03a006df70 2024-03-25T18:10:02.349 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.348+0000 7f039bfff640 1 -- 172.21.15.66:0/2331211309 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f03ac054ad0 con 0x7f03b41430c0 2024-03-25T18:10:02.349 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.348+0000 7f039bfff640 10 client.4495.objecter ms_dispatch 0x7f03b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:02.349 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.348+0000 7f039bfff640 3 client.4495.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:02.349 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.348+0000 7f039bfff640 3 client.4495.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:02.349 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.348+0000 7f039bfff640 20 client.4495.objecter dump_active .. 0 homeless 2024-03-25T18:10:02.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.352+0000 7f039bfff640 1 -- 172.21.15.66:0/2331211309 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f03ac0505e0 con 0x7f03b41430c0 2024-03-25T18:10:02.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.352+0000 7f039bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:02.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.352+0000 7f039bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:02.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.618+0000 7f03b980f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:02.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.618+0000 7f03b980f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:02.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.618+0000 7f03b980f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:10:02.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.618+0000 7f03b980f640 1 -- 172.21.15.66:0/2331211309 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03b40627b0 con 0x7f03b41430c0 2024-03-25T18:10:02.647 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.647+0000 7f039bfff640 1 -- 172.21.15.66:0/2331211309 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f03ac052510 con 0x7f03b41430c0 2024-03-25T18:10:02.647 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.647+0000 7f039bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:02.647 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.647+0000 7f039bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:02.647 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:10:02.647 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:02.650 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.649+0000 7f03b980f640 10 librados: watch_flush enter 2024-03-25T18:10:02.650 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.649+0000 7f03b980f640 10 librados: watch_flush exit 2024-03-25T18:10:02.650 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.649+0000 7f03b980f640 20 client.4495.objecter shutdown clearing up homeless session... 2024-03-25T18:10:02.650 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.649+0000 7f03b980f640 10 client.4495.objecter successfully canceled tick 2024-03-25T18:10:02.650 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.649+0000 7f03b980f640 1 -- 172.21.15.66:0/2331211309 >> v1:172.21.15.193:6800/2275676803 conn(0x7f03a006df70 legacy=0x7f03a0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:02.650 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.649+0000 7f03b980f640 10 monclient: shutdown 2024-03-25T18:10:02.650 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.649+0000 7f03b980f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:02.650 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.649+0000 7f03b980f640 1 -- 172.21.15.66:0/2331211309 >> v1:172.21.15.193:6789/0 conn(0x7f03b41430c0 legacy=0x7f03b41e2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:02.650 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.649+0000 7f03b980f640 1 -- 172.21.15.66:0/2331211309 shutdown_connections 2024-03-25T18:10:02.651 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.649+0000 7f03b980f640 1 -- 172.21.15.66:0/2331211309 >> 172.21.15.66:0/2331211309 conn(0x7f03b40b7ea0 msgr2=0x7f03b4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:02.651 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.649+0000 7f03b980f640 1 -- 172.21.15.66:0/2331211309 shutdown_connections 2024-03-25T18:10:02.651 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.649+0000 7f03b980f640 1 -- 172.21.15.66:0/2331211309 wait complete. 2024-03-25T18:10:02.651 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:02.649+0000 7f03b980f640 1 librados: shutdown 2024-03-25T18:10:02.662 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:09.468 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:09.576 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.575+0000 7f664b0d6640 10 monclient: get_monmap_and_config 2024-03-25T18:10:09.576 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.575+0000 7f664b0d6640 10 monclient: build_initial_monmap 2024-03-25T18:10:09.576 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.575+0000 7f664b0d6640 10 monclient: monmap: 2024-03-25T18:10:09.576 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:10:09.576 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:09.576 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:10:09.576416+0000 2024-03-25T18:10:09.576 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:10:09.576416+0000 2024-03-25T18:10:09.576 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:09.576 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:09.576 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:10:09.576 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:10:09.577 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:10:09.577 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:09.578 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.576+0000 7f664b0d6640 1 Processor -- start 2024-03-25T18:10:09.578 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.576+0000 7f664b0d6640 1 -- start start 2024-03-25T18:10:09.578 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.576+0000 7f664b0d6640 10 monclient: init 2024-03-25T18:10:09.578 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.577+0000 7f664b0d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:09.578 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.577+0000 7f664b0d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:10:09.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.577+0000 7f664b0d6640 10 monclient(hunting): picked mon.noname-c con 0x7f66440a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:10:09.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.577+0000 7f664b0d6640 10 monclient(hunting): picked mon.noname-a con 0x7f664413f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:10:09.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.577+0000 7f664b0d6640 10 monclient(hunting): picked mon.noname-b con 0x7f66441430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:10:09.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.577+0000 7f664b0d6640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6644142520 con 0x7f664413f710 2024-03-25T18:10:09.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.577+0000 7f664b0d6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66440b28a0 con 0x7f66440a98f0 2024-03-25T18:10:09.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.577+0000 7f664b0d6640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66440b2a30 con 0x7f66441430c0 2024-03-25T18:10:09.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.577+0000 7f664b0d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:09.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.577+0000 7f664b0d6640 10 monclient(hunting): authenticate will time out at 7358.636591s 2024-03-25T18:10:09.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.577+0000 7f6649673640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f66441430c0 0x7f664414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:59912/0 (socket says 172.21.15.66:59912) 2024-03-25T18:10:09.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.577+0000 7f6649673640 1 -- 172.21.15.66:0/2549732972 learned_addr learned my addr 172.21.15.66:0/2549732972 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:10:09.672 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.671+0000 7f66437fe640 1 -- 172.21.15.66:0/2549732972 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 111053712 0 0) 0x7f6644142520 con 0x7f664413f710 2024-03-25T18:10:09.672 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.671+0000 7f66437fe640 10 monclient(hunting): _negotiate 2024-03-25T18:10:09.672 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.671+0000 7f66437fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:09.672 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.671+0000 7f66437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:09.672 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.671+0000 7f66437fe640 10 monclient(hunting): my global_id is 4506 2024-03-25T18:10:09.672 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.671+0000 7f66437fe640 1 -- 172.21.15.66:0/2549732972 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6634003520 con 0x7f664413f710 2024-03-25T18:10:09.684 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.683+0000 7f66437fe640 1 -- 172.21.15.66:0/2549732972 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1006618250 0 0) 0x7f6634003520 con 0x7f664413f710 2024-03-25T18:10:09.684 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.683+0000 7f66437fe640 1 -- 172.21.15.66:0/2549732972 >> v1:172.21.15.66:6790/0 conn(0x7f66441430c0 legacy=0x7f664414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:09.684 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.683+0000 7f66437fe640 1 -- 172.21.15.66:0/2549732972 >> v1:172.21.15.193:6789/0 conn(0x7f66440a98f0 legacy=0x7f66440a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:09.684 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.683+0000 7f66437fe640 10 monclient: _finish_hunting 0 2024-03-25T18:10:09.684 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.683+0000 7f66437fe640 1 monclient: found mon.noname-a 2024-03-25T18:10:09.684 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.683+0000 7f66437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:09.684 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.683+0000 7f66437fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.66:6789/0 2024-03-25T18:10:09.684 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.683+0000 7f66437fe640 1 -- 172.21.15.66:0/2549732972 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66440b44c0 con 0x7f664413f710 2024-03-25T18:10:09.684 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.683+0000 7f66437fe640 10 monclient: _finish_auth 0 2024-03-25T18:10:09.684 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.683+0000 7f66437fe640 10 monclient: _check_auth_tickets 2024-03-25T18:10:09.685 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.683+0000 7f66437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:09.685 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.683+0000 7f664b0d6640 5 monclient: authenticate success, global_id 4506 2024-03-25T18:10:09.685 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.683+0000 7f66437fe640 1 -- 172.21.15.66:0/2549732972 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f6630001690 con 0x7f664413f710 2024-03-25T18:10:09.685 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.683+0000 7f66437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:09.685 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.683+0000 7f66437fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:10:09.685 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.683+0000 7f66437fe640 10 monclient: dump: 2024-03-25T18:10:09.685 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:09.685 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:09.685 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:09.685 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:09.685 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:09.685 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:09.685 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:09.685 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:09.685 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:09.685 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:09.686 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.683+0000 7f664b0d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:09.698 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.698+0000 7f66437fe640 1 -- 172.21.15.66:0/2549732972 <== mon.0 v1:172.21.15.66:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66300017f0 con 0x7f664413f710 2024-03-25T18:10:09.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.698+0000 7f66437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:09.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.698+0000 7f66437fe640 1 -- 172.21.15.66:0/2549732972 <== mon.0 v1:172.21.15.66:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f6630001c00 con 0x7f664413f710 2024-03-25T18:10:09.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.698+0000 7f664b0d6640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:09.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.698+0000 7f664b0d6640 4 set_mon_vals no callback set 2024-03-25T18:10:09.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.698+0000 7f66437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:09.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.698+0000 7f66437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:09.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.698+0000 7f66437fe640 10 monclient: dump: 2024-03-25T18:10:09.699 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:09.699 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:09.699 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:09.699 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:09.699 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:09.699 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:09.699 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:09.700 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:09.700 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:09.700 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:09.702 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.698+0000 7f664b0d6640 10 monclient: shutdown 2024-03-25T18:10:09.702 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.698+0000 7f664b0d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:09.702 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.698+0000 7f664b0d6640 1 -- 172.21.15.66:0/2549732972 >> v1:172.21.15.66:6789/0 conn(0x7f664413f710 legacy=0x7f66441429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:09.702 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.699+0000 7f664b0d6640 1 -- 172.21.15.66:0/2549732972 shutdown_connections 2024-03-25T18:10:09.702 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.699+0000 7f664b0d6640 1 -- 172.21.15.66:0/2549732972 >> 172.21.15.66:0/2549732972 conn(0x7f66440b7ea0 msgr2=0x7f66440b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:09.702 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.699+0000 7f664b0d6640 1 -- 172.21.15.66:0/2549732972 shutdown_connections 2024-03-25T18:10:09.702 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.699+0000 7f664b0d6640 1 -- 172.21.15.66:0/2549732972 wait complete. 2024-03-25T18:10:09.702 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.699+0000 7f664b0d6640 10 monclient: build_initial_monmap 2024-03-25T18:10:09.702 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.699+0000 7f664b0d6640 10 monclient: monmap: 2024-03-25T18:10:09.702 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:10:09.703 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:09.703 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:10:09.703 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:10:09.703 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:09.703 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:10:09.703 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:10:09.703 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:10:09.703 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:10:09.703 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:09.703 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.700+0000 7f664b0d6640 1 librados: starting msgr at 2024-03-25T18:10:09.703 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.700+0000 7f664b0d6640 1 librados: starting objecter 2024-03-25T18:10:09.703 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.700+0000 7f664b0d6640 1 Processor -- start 2024-03-25T18:10:09.703 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.700+0000 7f664b0d6640 1 -- start start 2024-03-25T18:10:09.703 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.700+0000 7f664b0d6640 1 librados: setting wanted keys 2024-03-25T18:10:09.703 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.700+0000 7f664b0d6640 1 librados: calling monclient init 2024-03-25T18:10:09.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.700+0000 7f664b0d6640 10 monclient: init 2024-03-25T18:10:09.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.701+0000 7f664b0d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:09.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.701+0000 7f664b0d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:10:09.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.701+0000 7f664b0d6640 10 monclient(hunting): picked mon.noname-a con 0x7f66440a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:10:09.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.701+0000 7f664b0d6640 10 monclient(hunting): picked mon.noname-c con 0x7f664413f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:10:09.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.701+0000 7f664b0d6640 10 monclient(hunting): picked mon.noname-b con 0x7f66441430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:10:09.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.701+0000 7f664b0d6640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66440abd70 con 0x7f66440a98f0 2024-03-25T18:10:09.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.701+0000 7f664b0d6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66441d7710 con 0x7f66441430c0 2024-03-25T18:10:09.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.701+0000 7f664b0d6640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66441d8340 con 0x7f664413f710 2024-03-25T18:10:09.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.701+0000 7f664b0d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:09.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.701+0000 7f664b0d6640 10 monclient(hunting): authenticate will time out at 7358.760304s 2024-03-25T18:10:09.705 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.702+0000 7f6649673640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f66441430c0 0x7f66441d4860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:56354/0 (socket says 172.21.15.66:56354) 2024-03-25T18:10:09.705 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.702+0000 7f6649673640 1 -- 172.21.15.66:0/517296873 learned_addr learned my addr 172.21.15.66:0/517296873 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:10:09.705 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.703+0000 7f66417fa640 10 client.?.objecter ms_handle_connect 0x7f66441430c0 2024-03-25T18:10:09.705 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.703+0000 7f66417fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:09.713 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.713+0000 7f66417fa640 10 client.?.objecter ms_handle_connect 0x7f664413f710 2024-03-25T18:10:09.713 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.713+0000 7f66417fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:09.757 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:06:40.738 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:09.768 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:06:40.749 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:09.785 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.785+0000 7f66417fa640 10 client.?.objecter ms_handle_connect 0x7f66440a98f0 2024-03-25T18:10:09.785 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.785+0000 7f66417fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:09.806 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:06:40.787 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:09.806 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.806+0000 7f66417fa640 1 -- 172.21.15.66:0/517296873 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2799116448 0 0) 0x7f66440abd70 con 0x7f66440a98f0 2024-03-25T18:10:09.806 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.806+0000 7f66417fa640 10 monclient(hunting): _negotiate 2024-03-25T18:10:09.807 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.806+0000 7f66417fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:09.807 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.806+0000 7f66417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:09.807 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.806+0000 7f66417fa640 10 monclient(hunting): my global_id is 4509 2024-03-25T18:10:09.807 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.806+0000 7f66417fa640 1 -- 172.21.15.66:0/517296873 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6628004eb0 con 0x7f66440a98f0 2024-03-25T18:10:09.820 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:06:40.779 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:09.823 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:06:40.782 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:09.829 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:06:40.810 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:09.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.830+0000 7f66417fa640 1 -- 172.21.15.66:0/517296873 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3843039518 0 0) 0x7f66441d7710 con 0x7f66441430c0 2024-03-25T18:10:09.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.830+0000 7f66417fa640 10 monclient(hunting): _negotiate 2024-03-25T18:10:09.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.830+0000 7f66417fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:09.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.830+0000 7f66417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:09.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.830+0000 7f66417fa640 10 monclient(hunting): my global_id is 4501 2024-03-25T18:10:09.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.830+0000 7f66417fa640 1 -- 172.21.15.66:0/517296873 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66440abd70 con 0x7f66441430c0 2024-03-25T18:10:09.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.830+0000 7f66417fa640 1 -- 172.21.15.66:0/517296873 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4196579572 0 0) 0x7f66441d8340 con 0x7f664413f710 2024-03-25T18:10:09.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.830+0000 7f66417fa640 10 monclient(hunting): _negotiate 2024-03-25T18:10:09.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.830+0000 7f66417fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:09.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.830+0000 7f66417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:09.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.830+0000 7f66417fa640 10 monclient(hunting): my global_id is 4502 2024-03-25T18:10:09.832 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.830+0000 7f66417fa640 1 -- 172.21.15.66:0/517296873 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66441d7710 con 0x7f664413f710 2024-03-25T18:10:09.833 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:06:40.813 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:09.853 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.853+0000 7f66417fa640 1 -- 172.21.15.66:0/517296873 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3054929079 0 0) 0x7f66440abd70 con 0x7f66441430c0 2024-03-25T18:10:09.853 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.853+0000 7f66417fa640 1 -- 172.21.15.66:0/517296873 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66441d8340 con 0x7f66441430c0 2024-03-25T18:10:09.853 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.853+0000 7f66417fa640 1 -- 172.21.15.66:0/517296873 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f663c001b50 con 0x7f66441430c0 2024-03-25T18:10:09.853 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.853+0000 7f66417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:09.853 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.853+0000 7f66417fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:09.853 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.853+0000 7f66417fa640 10 monclient(hunting): dump: 2024-03-25T18:10:09.853 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:09.854 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:09.854 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:09.854 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:09.854 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:09.854 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:09.854 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:09.854 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:09.854 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:09.854 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:09.854 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.853+0000 7f66417fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:09.861 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:06:40.819 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:09.866 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:06:40.847 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:09.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.873+0000 7f66417fa640 1 -- 172.21.15.66:0/517296873 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1416969898 0 0) 0x7f6628004eb0 con 0x7f66440a98f0 2024-03-25T18:10:09.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.873+0000 7f66417fa640 1 -- 172.21.15.66:0/517296873 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66440abd70 con 0x7f66440a98f0 2024-03-25T18:10:09.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.873+0000 7f66417fa640 1 -- 172.21.15.66:0/517296873 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f66380018c0 con 0x7f66440a98f0 2024-03-25T18:10:09.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.873+0000 7f66417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:09.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.873+0000 7f66417fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:09.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.873+0000 7f66417fa640 10 monclient(hunting): dump: 2024-03-25T18:10:09.874 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:09.875 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:09.875 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:09.875 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:09.875 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:09.875 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:09.875 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:09.875 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:09.875 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:09.875 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:09.876 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.875+0000 7f66417fa640 1 -- 172.21.15.66:0/517296873 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1268778761 0 0) 0x7f66441d7710 con 0x7f664413f710 2024-03-25T18:10:09.876 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.875+0000 7f66417fa640 1 -- 172.21.15.66:0/517296873 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6628004eb0 con 0x7f664413f710 2024-03-25T18:10:09.876 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.875+0000 7f66417fa640 1 -- 172.21.15.66:0/517296873 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f6630001bc0 con 0x7f664413f710 2024-03-25T18:10:09.876 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.875+0000 7f66417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:09.876 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.875+0000 7f66417fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:09.876 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.875+0000 7f66417fa640 10 monclient(hunting): dump: 2024-03-25T18:10:09.876 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:09.876 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:09.876 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:09.876 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:09.876 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:09.876 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:09.876 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:09.876 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:09.877 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:09.877 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:09.881 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.880+0000 7f66417fa640 1 -- 172.21.15.66:0/517296873 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 16276722 0 0) 0x7f66441d8340 con 0x7f66441430c0 2024-03-25T18:10:09.881 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.880+0000 7f66417fa640 1 -- 172.21.15.66:0/517296873 >> v1:172.21.15.66:6790/0 conn(0x7f664413f710 legacy=0x7f66441d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:09.881 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.880+0000 7f66417fa640 1 -- 172.21.15.66:0/517296873 >> v1:172.21.15.66:6789/0 conn(0x7f66440a98f0 legacy=0x7f6644137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:09.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.880+0000 7f66417fa640 10 monclient: _finish_hunting 0 2024-03-25T18:10:09.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.880+0000 7f66417fa640 1 monclient: found mon.b 2024-03-25T18:10:09.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.880+0000 7f66417fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:09.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.880+0000 7f66417fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:10:09.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.880+0000 7f66417fa640 1 -- 172.21.15.66:0/517296873 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66441d84d0 con 0x7f66441430c0 2024-03-25T18:10:09.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.880+0000 7f66417fa640 10 monclient: _finish_auth 0 2024-03-25T18:10:09.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.880+0000 7f66417fa640 10 monclient: _check_auth_tickets 2024-03-25T18:10:09.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.880+0000 7f66417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:09.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.880+0000 7f664b0d6640 5 monclient: authenticate success, global_id 4501 2024-03-25T18:10:09.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.880+0000 7f664b0d6640 10 monclient: _renew_subs 2024-03-25T18:10:09.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.880+0000 7f664b0d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:10:09.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.880+0000 7f664b0d6640 1 -- 172.21.15.66:0/517296873 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66441d41e0 con 0x7f66441430c0 2024-03-25T18:10:09.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.881+0000 7f664b0d6640 10 client.4501.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:09.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.881+0000 7f664b0d6640 10 monclient: _renew_subs 2024-03-25T18:10:09.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.881+0000 7f664b0d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:10:09.883 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.881+0000 7f664b0d6640 1 -- 172.21.15.66:0/517296873 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f66441d5150 con 0x7f66441430c0 2024-03-25T18:10:09.883 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.881+0000 7f664b0d6640 1 librados: init done 2024-03-25T18:10:09.883 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.881+0000 7f664b0d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:09.883 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.882+0000 7f664b0d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:09.883 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.882+0000 7f664b0d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:10:09.883 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:09.882+0000 7f664b0d6640 1 -- 172.21.15.66:0/517296873 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6644074cc0 con 0x7f66441430c0 2024-03-25T18:10:10.189 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.188+0000 7f66417fa640 1 -- 172.21.15.66:0/517296873 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f663c001f80 con 0x7f66441430c0 2024-03-25T18:10:10.189 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.188+0000 7f66417fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:10.189 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.188+0000 7f66417fa640 1 -- 172.21.15.66:0/517296873 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f663c004d00 con 0x7f66441430c0 2024-03-25T18:10:10.189 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.188+0000 7f66417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:10.189 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.188+0000 7f66417fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:10.189 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.188+0000 7f66417fa640 10 monclient: dump: 2024-03-25T18:10:10.189 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:10.189 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:10.190 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:10.190 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:10.190 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:10.190 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:10.190 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:10.190 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:10.190 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:10.190 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:10.190 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.188+0000 7f6642ffd640 4 set_mon_vals no callback set 2024-03-25T18:10:10.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.189+0000 7f66417fa640 1 -- 172.21.15.66:0/517296873 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f663c01acd0 con 0x7f66441430c0 2024-03-25T18:10:10.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.190+0000 7f6648e72640 10 monclient: get_auth_request con 0x7f662806e0b0 auth_method 0 2024-03-25T18:10:10.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.190+0000 7f66417fa640 10 client.4501.objecter ms_handle_connect 0x7f662806e0b0 2024-03-25T18:10:10.195 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.195+0000 7f66417fa640 1 -- 172.21.15.66:0/517296873 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f663c054a30 con 0x7f66441430c0 2024-03-25T18:10:10.195 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.195+0000 7f66417fa640 10 client.4501.objecter ms_dispatch 0x7f6644000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:10.195 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.195+0000 7f66417fa640 3 client.4501.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:10.195 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.195+0000 7f66417fa640 3 client.4501.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:10.195 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.195+0000 7f66417fa640 20 client.4501.objecter dump_active .. 0 homeless 2024-03-25T18:10:10.196 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.195+0000 7f66417fa640 1 -- 172.21.15.66:0/517296873 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f663c050370 con 0x7f66441430c0 2024-03-25T18:10:10.196 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.195+0000 7f66417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:10.196 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.195+0000 7f66417fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:10.449 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.448+0000 7f664b0d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:10.449 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.448+0000 7f664b0d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:10.449 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.448+0000 7f664b0d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:10:10.449 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.448+0000 7f664b0d6640 1 -- 172.21.15.66:0/517296873 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66441d78a0 con 0x7f66441430c0 2024-03-25T18:10:10.474 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.473+0000 7f66417fa640 1 -- 172.21.15.66:0/517296873 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f663c052470 con 0x7f66441430c0 2024-03-25T18:10:10.474 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.473+0000 7f66417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:10.474 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.473+0000 7f66417fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:10.474 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:10:10.474 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:10.477 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.475+0000 7f664b0d6640 10 librados: watch_flush enter 2024-03-25T18:10:10.477 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.476+0000 7f664b0d6640 10 librados: watch_flush exit 2024-03-25T18:10:10.477 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.476+0000 7f664b0d6640 20 client.4501.objecter shutdown clearing up homeless session... 2024-03-25T18:10:10.477 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.476+0000 7f664b0d6640 10 client.4501.objecter successfully canceled tick 2024-03-25T18:10:10.477 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.476+0000 7f664b0d6640 1 -- 172.21.15.66:0/517296873 >> v1:172.21.15.193:6800/2275676803 conn(0x7f662806e0b0 legacy=0x7f6628070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:10.477 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.476+0000 7f664b0d6640 10 monclient: shutdown 2024-03-25T18:10:10.477 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.476+0000 7f664b0d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:10.477 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.476+0000 7f664b0d6640 1 -- 172.21.15.66:0/517296873 >> v1:172.21.15.193:6789/0 conn(0x7f66441430c0 legacy=0x7f66441d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:10.477 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.476+0000 7f664b0d6640 1 -- 172.21.15.66:0/517296873 shutdown_connections 2024-03-25T18:10:10.477 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.476+0000 7f664b0d6640 1 -- 172.21.15.66:0/517296873 >> 172.21.15.66:0/517296873 conn(0x7f66440b7ea0 msgr2=0x7f6644074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:10.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.476+0000 7f664b0d6640 1 -- 172.21.15.66:0/517296873 shutdown_connections 2024-03-25T18:10:10.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.476+0000 7f664b0d6640 1 -- 172.21.15.66:0/517296873 wait complete. 2024-03-25T18:10:10.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:10.476+0000 7f664b0d6640 1 librados: shutdown 2024-03-25T18:10:10.490 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:17.297 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:17.418 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.417+0000 7f29c42e2640 10 monclient: get_monmap_and_config 2024-03-25T18:10:17.418 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.417+0000 7f29c42e2640 10 monclient: build_initial_monmap 2024-03-25T18:10:17.418 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.417+0000 7f29c42e2640 10 monclient: monmap: 2024-03-25T18:10:17.418 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:10:17.419 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:17.419 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:10:17.418100+0000 2024-03-25T18:10:17.419 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:10:17.418100+0000 2024-03-25T18:10:17.419 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:17.419 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:17.419 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:10:17.420 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:10:17.420 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:10:17.420 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:17.421 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.418+0000 7f29c42e2640 1 Processor -- start 2024-03-25T18:10:17.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.418+0000 7f29c42e2640 1 -- start start 2024-03-25T18:10:17.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.418+0000 7f29c42e2640 10 monclient: init 2024-03-25T18:10:17.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.419+0000 7f29c42e2640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:17.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.419+0000 7f29c42e2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:10:17.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.419+0000 7f29c42e2640 10 monclient(hunting): picked mon.noname-b con 0x7f29bc070380 addr v1:172.21.15.66:6790/0 2024-03-25T18:10:17.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.419+0000 7f29c42e2640 10 monclient(hunting): picked mon.noname-c con 0x7f29bc070760 addr v1:172.21.15.193:6789/0 2024-03-25T18:10:17.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.419+0000 7f29c42e2640 10 monclient(hunting): picked mon.noname-a con 0x7f29bc0ae470 addr v1:172.21.15.66:6789/0 2024-03-25T18:10:17.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.419+0000 7f29c42e2640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29bc13dd40 con 0x7f29bc0ae470 2024-03-25T18:10:17.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.419+0000 7f29c42e2640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29bc071740 con 0x7f29bc070760 2024-03-25T18:10:17.423 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.419+0000 7f29c42e2640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29bc0725d0 con 0x7f29bc070380 2024-03-25T18:10:17.423 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.419+0000 7f29c42e2640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:17.423 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.419+0000 7f29c42e2640 10 monclient(hunting): authenticate will time out at 7366.478913s 2024-03-25T18:10:17.426 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.425+0000 7f29c187d640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f29bc070760 0x7f29bc13cf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:37950/0 (socket says 172.21.15.66:37950) 2024-03-25T18:10:17.426 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.425+0000 7f29c187d640 1 -- 172.21.15.66:0/641909728 learned_addr learned my addr 172.21.15.66:0/641909728 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:10:17.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.446+0000 7f29c107c640 1 -- 172.21.15.66:0/641909728 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728455724 0 0) 0x7f29bc071740 con 0x7f29bc070760 2024-03-25T18:10:17.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.446+0000 7f29c107c640 10 monclient(hunting): _negotiate 2024-03-25T18:10:17.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.446+0000 7f29c107c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:17.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.446+0000 7f29c107c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:17.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.446+0000 7f29c107c640 10 monclient(hunting): my global_id is 4504 2024-03-25T18:10:17.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.446+0000 7f29c107c640 1 -- 172.21.15.66:0/641909728 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29a8003520 con 0x7f29bc070760 2024-03-25T18:10:17.456 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.456+0000 7f29c107c640 1 -- 172.21.15.66:0/641909728 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3369921452 0 0) 0x7f29bc13dd40 con 0x7f29bc0ae470 2024-03-25T18:10:17.456 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.456+0000 7f29c107c640 10 monclient(hunting): _negotiate 2024-03-25T18:10:17.456 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.456+0000 7f29c107c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:17.457 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.456+0000 7f29c107c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:17.457 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.456+0000 7f29c107c640 10 monclient(hunting): my global_id is 4512 2024-03-25T18:10:17.457 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.456+0000 7f29c107c640 1 -- 172.21.15.66:0/641909728 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29bc071740 con 0x7f29bc0ae470 2024-03-25T18:10:17.459 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.458+0000 7f29c107c640 1 -- 172.21.15.66:0/641909728 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 406604275 0 0) 0x7f29a8003520 con 0x7f29bc070760 2024-03-25T18:10:17.459 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.458+0000 7f29c107c640 1 -- 172.21.15.66:0/641909728 >> v1:172.21.15.66:6790/0 conn(0x7f29bc070380 legacy=0x7f29bc140810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:17.460 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.458+0000 7f29c107c640 1 -- 172.21.15.66:0/641909728 >> v1:172.21.15.66:6789/0 conn(0x7f29bc0ae470 legacy=0x7f29bc13d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:17.460 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.458+0000 7f29c107c640 10 monclient: _finish_hunting 0 2024-03-25T18:10:17.460 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.458+0000 7f29c107c640 1 monclient: found mon.noname-c 2024-03-25T18:10:17.460 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.458+0000 7f29c107c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:17.460 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.458+0000 7f29c107c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:10:17.460 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.458+0000 7f29c107c640 1 -- 172.21.15.66:0/641909728 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29bc0728b0 con 0x7f29bc070760 2024-03-25T18:10:17.460 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.458+0000 7f29c107c640 10 monclient: _finish_auth 0 2024-03-25T18:10:17.460 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.458+0000 7f29c107c640 10 monclient: _check_auth_tickets 2024-03-25T18:10:17.460 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.458+0000 7f29c107c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:17.460 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.458+0000 7f29c107c640 1 -- 172.21.15.66:0/641909728 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f29ac001690 con 0x7f29bc070760 2024-03-25T18:10:17.461 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.458+0000 7f29c107c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:17.461 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.458+0000 7f29c107c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:17.461 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.458+0000 7f29c107c640 10 monclient: dump: 2024-03-25T18:10:17.461 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:17.461 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:17.461 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:17.461 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:17.461 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:17.461 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:17.461 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:17.461 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:17.461 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:17.462 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:17.462 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.458+0000 7f29c42e2640 5 monclient: authenticate success, global_id 4504 2024-03-25T18:10:17.462 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.458+0000 7f29c42e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:17.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.496+0000 7f29c107c640 1 -- 172.21.15.66:0/641909728 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29ac001870 con 0x7f29bc070760 2024-03-25T18:10:17.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.496+0000 7f29c107c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:17.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.496+0000 7f29c107c640 1 -- 172.21.15.66:0/641909728 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f29ac001c80 con 0x7f29bc070760 2024-03-25T18:10:17.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.496+0000 7f29c42e2640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:17.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.496+0000 7f29c42e2640 4 set_mon_vals no callback set 2024-03-25T18:10:17.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.496+0000 7f29c107c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:17.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.496+0000 7f29c107c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:17.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.496+0000 7f29c107c640 10 monclient: dump: 2024-03-25T18:10:17.498 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:17.498 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:17.498 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:17.498 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:17.499 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:17.499 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:17.499 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:17.499 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:17.499 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:17.499 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:17.502 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.497+0000 7f29c42e2640 10 monclient: shutdown 2024-03-25T18:10:17.502 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.497+0000 7f29c42e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:17.502 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.497+0000 7f29c42e2640 1 -- 172.21.15.66:0/641909728 >> v1:172.21.15.193:6789/0 conn(0x7f29bc070760 legacy=0x7f29bc13cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:17.502 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.497+0000 7f29c42e2640 1 -- 172.21.15.66:0/641909728 shutdown_connections 2024-03-25T18:10:17.502 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.497+0000 7f29c42e2640 1 -- 172.21.15.66:0/641909728 >> 172.21.15.66:0/641909728 conn(0x7f29bc06b630 msgr2=0x7f29bc0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:17.502 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.497+0000 7f29c42e2640 1 -- 172.21.15.66:0/641909728 shutdown_connections 2024-03-25T18:10:17.502 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.497+0000 7f29c42e2640 1 -- 172.21.15.66:0/641909728 wait complete. 2024-03-25T18:10:17.502 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.498+0000 7f29c42e2640 10 monclient: build_initial_monmap 2024-03-25T18:10:17.502 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.498+0000 7f29c42e2640 10 monclient: monmap: 2024-03-25T18:10:17.503 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:10:17.503 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:17.503 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:10:17.503 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:10:17.503 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:17.503 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:10:17.503 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:10:17.503 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:10:17.503 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:10:17.503 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:17.503 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.498+0000 7f29c42e2640 1 librados: starting msgr at 2024-03-25T18:10:17.503 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.498+0000 7f29c42e2640 1 librados: starting objecter 2024-03-25T18:10:17.504 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.499+0000 7f29c42e2640 1 Processor -- start 2024-03-25T18:10:17.504 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.499+0000 7f29c42e2640 1 -- start start 2024-03-25T18:10:17.504 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.499+0000 7f29c42e2640 1 librados: setting wanted keys 2024-03-25T18:10:17.504 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.499+0000 7f29c42e2640 1 librados: calling monclient init 2024-03-25T18:10:17.504 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.499+0000 7f29c42e2640 10 monclient: init 2024-03-25T18:10:17.504 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.499+0000 7f29c42e2640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:17.504 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.499+0000 7f29c42e2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:10:17.504 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.499+0000 7f29c42e2640 10 monclient(hunting): picked mon.noname-a con 0x7f29bc070380 addr v1:172.21.15.66:6789/0 2024-03-25T18:10:17.504 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.499+0000 7f29c42e2640 10 monclient(hunting): picked mon.noname-c con 0x7f29bc070760 addr v1:172.21.15.66:6790/0 2024-03-25T18:10:17.504 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.499+0000 7f29c42e2640 10 monclient(hunting): picked mon.noname-b con 0x7f29bc0ae470 addr v1:172.21.15.193:6789/0 2024-03-25T18:10:17.504 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.499+0000 7f29c42e2640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29bc1d7f60 con 0x7f29bc070380 2024-03-25T18:10:17.505 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.499+0000 7f29c42e2640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29bc1d7b20 con 0x7f29bc0ae470 2024-03-25T18:10:17.505 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.499+0000 7f29c42e2640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29bc1d7cb0 con 0x7f29bc070760 2024-03-25T18:10:17.505 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.499+0000 7f29c42e2640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:17.505 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.499+0000 7f29c42e2640 10 monclient(hunting): authenticate will time out at 7366.558871s 2024-03-25T18:10:17.510 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.509+0000 7f29c287f640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f29bc0ae470 0x7f29bc06e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:38024/0 (socket says 172.21.15.66:38024) 2024-03-25T18:10:17.510 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.509+0000 7f29c287f640 1 -- 172.21.15.66:0/2817357269 learned_addr learned my addr 172.21.15.66:0/2817357269 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:10:17.513 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.513+0000 7f29b2ffd640 10 client.?.objecter ms_handle_connect 0x7f29bc0ae470 2024-03-25T18:10:17.513 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.513+0000 7f29b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:17.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.515+0000 7f29b2ffd640 10 client.?.objecter ms_handle_connect 0x7f29bc070380 2024-03-25T18:10:17.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.515+0000 7f29b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:17.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.520+0000 7f29b2ffd640 1 -- 172.21.15.66:0/2817357269 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2364397784 0 0) 0x7f29bc1d7b20 con 0x7f29bc0ae470 2024-03-25T18:10:17.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.520+0000 7f29b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:17.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.520+0000 7f29b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:17.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.520+0000 7f29b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:17.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.520+0000 7f29b2ffd640 10 monclient(hunting): my global_id is 4507 2024-03-25T18:10:17.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.520+0000 7f29b2ffd640 1 -- 172.21.15.66:0/2817357269 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29a0004eb0 con 0x7f29bc0ae470 2024-03-25T18:10:17.523 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.522+0000 7f29b2ffd640 10 client.?.objecter ms_handle_connect 0x7f29bc070760 2024-03-25T18:10:17.523 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.522+0000 7f29b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:17.532 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.531+0000 7f29b2ffd640 1 -- 172.21.15.66:0/2817357269 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1177473480 0 0) 0x7f29a0004eb0 con 0x7f29bc0ae470 2024-03-25T18:10:17.532 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.531+0000 7f29b2ffd640 1 -- 172.21.15.66:0/2817357269 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29bc1d7b20 con 0x7f29bc0ae470 2024-03-25T18:10:17.532 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.531+0000 7f29b2ffd640 1 -- 172.21.15.66:0/2817357269 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f29b4001c90 con 0x7f29bc0ae470 2024-03-25T18:10:17.532 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.531+0000 7f29b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:17.532 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.531+0000 7f29b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:17.532 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.531+0000 7f29b2ffd640 10 monclient(hunting): dump: 2024-03-25T18:10:17.532 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:17.532 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:17.533 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:17.533 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:17.533 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:17.533 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:17.533 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:17.533 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:17.533 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:17.533 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:17.533 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.531+0000 7f29b2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:17.539 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.538+0000 7f29b2ffd640 1 -- 172.21.15.66:0/2817357269 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367307709 0 0) 0x7f29bc1d7f60 con 0x7f29bc070380 2024-03-25T18:10:17.539 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.538+0000 7f29b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:17.539 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.538+0000 7f29b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:17.539 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.538+0000 7f29b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:17.539 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.538+0000 7f29b2ffd640 10 monclient(hunting): my global_id is 4515 2024-03-25T18:10:17.540 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.538+0000 7f29b2ffd640 1 -- 172.21.15.66:0/2817357269 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29a0004eb0 con 0x7f29bc070380 2024-03-25T18:10:17.544 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.542+0000 7f29b2ffd640 1 -- 172.21.15.66:0/2817357269 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 309228763 0 0) 0x7f29bc1d7b20 con 0x7f29bc0ae470 2024-03-25T18:10:17.544 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.543+0000 7f29b2ffd640 1 -- 172.21.15.66:0/2817357269 >> v1:172.21.15.66:6790/0 conn(0x7f29bc070760 legacy=0x7f29bc132d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:17.545 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.543+0000 7f29b2ffd640 1 -- 172.21.15.66:0/2817357269 >> v1:172.21.15.66:6789/0 conn(0x7f29bc070380 legacy=0x7f29bc1d7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:17.545 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.543+0000 7f29b2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:10:17.545 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.543+0000 7f29b2ffd640 1 monclient: found mon.b 2024-03-25T18:10:17.545 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.543+0000 7f29b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:17.545 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.543+0000 7f29b2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:10:17.545 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.543+0000 7f29b2ffd640 1 -- 172.21.15.66:0/2817357269 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29bc133420 con 0x7f29bc0ae470 2024-03-25T18:10:17.545 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.543+0000 7f29b2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:10:17.545 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.543+0000 7f29b2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:10:17.545 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.543+0000 7f29b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:17.545 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.543+0000 7f29c42e2640 5 monclient: authenticate success, global_id 4507 2024-03-25T18:10:17.545 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.543+0000 7f29c42e2640 10 monclient: _renew_subs 2024-03-25T18:10:17.546 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.543+0000 7f29c42e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:10:17.546 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.543+0000 7f29c42e2640 1 -- 172.21.15.66:0/2817357269 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29bc133670 con 0x7f29bc0ae470 2024-03-25T18:10:17.546 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.543+0000 7f29c42e2640 10 client.4507.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:17.546 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.543+0000 7f29c42e2640 10 monclient: _renew_subs 2024-03-25T18:10:17.546 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.543+0000 7f29c42e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:10:17.546 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.543+0000 7f29c42e2640 1 -- 172.21.15.66:0/2817357269 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29bc138390 con 0x7f29bc0ae470 2024-03-25T18:10:17.546 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.543+0000 7f29c42e2640 1 librados: init done 2024-03-25T18:10:17.546 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.544+0000 7f29c42e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:17.546 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.544+0000 7f29c42e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:17.546 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.544+0000 7f29c42e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:10:17.546 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.544+0000 7f29c42e2640 1 -- 172.21.15.66:0/2817357269 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29bc071b00 con 0x7f29bc0ae470 2024-03-25T18:10:17.812 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.812+0000 7f29b2ffd640 1 -- 172.21.15.66:0/2817357269 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29b4003990 con 0x7f29bc0ae470 2024-03-25T18:10:17.812 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.812+0000 7f29b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:17.813 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.812+0000 7f29b2ffd640 1 -- 172.21.15.66:0/2817357269 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f29b4003da0 con 0x7f29bc0ae470 2024-03-25T18:10:17.813 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.812+0000 7f29b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:17.813 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.812+0000 7f29b2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:17.813 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.812+0000 7f29b2ffd640 10 monclient: dump: 2024-03-25T18:10:17.813 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:17.813 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:17.813 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:17.813 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:17.813 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:17.813 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:17.814 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:17.814 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:17.814 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:17.814 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:17.814 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.812+0000 7f29c087b640 4 set_mon_vals no callback set 2024-03-25T18:10:17.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.812+0000 7f29b2ffd640 1 -- 172.21.15.66:0/2817357269 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f29b4003fc0 con 0x7f29bc0ae470 2024-03-25T18:10:17.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.813+0000 7f29c207e640 10 monclient: get_auth_request con 0x7f29a006dd90 auth_method 0 2024-03-25T18:10:17.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.813+0000 7f29b2ffd640 10 client.4507.objecter ms_handle_connect 0x7f29a006dd90 2024-03-25T18:10:17.817 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.817+0000 7f29b2ffd640 1 -- 172.21.15.66:0/2817357269 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f29b4054a70 con 0x7f29bc0ae470 2024-03-25T18:10:17.818 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.817+0000 7f29b2ffd640 10 client.4507.objecter ms_dispatch 0x7f29bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:17.818 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.817+0000 7f29b2ffd640 3 client.4507.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:17.818 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.817+0000 7f29b2ffd640 3 client.4507.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:17.818 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.817+0000 7f29b2ffd640 20 client.4507.objecter dump_active .. 0 homeless 2024-03-25T18:10:17.818 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.818+0000 7f29b2ffd640 1 -- 172.21.15.66:0/2817357269 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f29b4050580 con 0x7f29bc0ae470 2024-03-25T18:10:17.819 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.818+0000 7f29b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:17.819 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:17.818+0000 7f29b2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:18.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:18.063+0000 7f29c42e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:18.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:18.063+0000 7f29c42e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:18.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:18.063+0000 7f29c42e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:10:18.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:18.063+0000 7f29c42e2640 1 -- 172.21.15.66:0/2817357269 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29bc0aa140 con 0x7f29bc0ae470 2024-03-25T18:10:18.093 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:18.092+0000 7f29b2ffd640 1 -- 172.21.15.66:0/2817357269 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f29b40524b0 con 0x7f29bc0ae470 2024-03-25T18:10:18.093 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:18.092+0000 7f29b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:18.093 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:18.092+0000 7f29b2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:18.093 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:10:18.093 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:18.096 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:18.094+0000 7f29c42e2640 10 librados: watch_flush enter 2024-03-25T18:10:18.096 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:18.094+0000 7f29c42e2640 10 librados: watch_flush exit 2024-03-25T18:10:18.096 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:18.094+0000 7f29c42e2640 20 client.4507.objecter shutdown clearing up homeless session... 2024-03-25T18:10:18.096 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:18.094+0000 7f29c42e2640 10 client.4507.objecter successfully canceled tick 2024-03-25T18:10:18.096 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:18.094+0000 7f29c42e2640 1 -- 172.21.15.66:0/2817357269 >> v1:172.21.15.193:6800/2275676803 conn(0x7f29a006dd90 legacy=0x7f29a0070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:18.096 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:18.094+0000 7f29c42e2640 10 monclient: shutdown 2024-03-25T18:10:18.096 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:18.094+0000 7f29c42e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:18.096 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:18.094+0000 7f29c42e2640 1 -- 172.21.15.66:0/2817357269 >> v1:172.21.15.193:6789/0 conn(0x7f29bc0ae470 legacy=0x7f29bc06e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:18.096 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:18.094+0000 7f29c42e2640 1 -- 172.21.15.66:0/2817357269 shutdown_connections 2024-03-25T18:10:18.096 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:18.094+0000 7f29c42e2640 1 -- 172.21.15.66:0/2817357269 >> 172.21.15.66:0/2817357269 conn(0x7f29bc06b630 msgr2=0x7f29bc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:18.096 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:18.094+0000 7f29c42e2640 1 -- 172.21.15.66:0/2817357269 shutdown_connections 2024-03-25T18:10:18.097 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:18.094+0000 7f29c42e2640 1 -- 172.21.15.66:0/2817357269 wait complete. 2024-03-25T18:10:18.097 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:18.094+0000 7f29c42e2640 1 librados: shutdown 2024-03-25T18:10:18.106 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:24.913 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:25.033 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.032+0000 7f83448e6640 10 monclient: get_monmap_and_config 2024-03-25T18:10:25.033 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.032+0000 7f83448e6640 10 monclient: build_initial_monmap 2024-03-25T18:10:25.033 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.032+0000 7f83448e6640 10 monclient: monmap: 2024-03-25T18:10:25.033 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:10:25.033 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:25.033 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:10:25.033084+0000 2024-03-25T18:10:25.033 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:10:25.033084+0000 2024-03-25T18:10:25.033 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:25.033 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:25.034 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:10:25.034 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:10:25.034 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:10:25.034 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:25.035 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.033+0000 7f83448e6640 1 Processor -- start 2024-03-25T18:10:25.035 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.033+0000 7f83448e6640 1 -- start start 2024-03-25T18:10:25.035 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.033+0000 7f83448e6640 10 monclient: init 2024-03-25T18:10:25.035 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.034+0000 7f83448e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:25.035 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.034+0000 7f83448e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:10:25.036 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.034+0000 7f83448e6640 10 monclient(hunting): picked mon.noname-c con 0x7f83400a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:10:25.036 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.034+0000 7f83448e6640 10 monclient(hunting): picked mon.noname-a con 0x7f834013f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:10:25.036 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.034+0000 7f83448e6640 10 monclient(hunting): picked mon.noname-b con 0x7f83401430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:10:25.036 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.034+0000 7f83448e6640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8340142520 con 0x7f834013f710 2024-03-25T18:10:25.036 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.034+0000 7f83448e6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83400b28a0 con 0x7f83400a98f0 2024-03-25T18:10:25.036 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.034+0000 7f83448e6640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83400b2a30 con 0x7f83401430c0 2024-03-25T18:10:25.036 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.034+0000 7f83448e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:25.036 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.034+0000 7f83448e6640 10 monclient(hunting): authenticate will time out at 7374.093430s 2024-03-25T18:10:25.039 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.038+0000 7f833e59c640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f83400a98f0 0x7f83400a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:35268/0 (socket says 172.21.15.66:35268) 2024-03-25T18:10:25.039 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.038+0000 7f833e59c640 1 -- 172.21.15.66:0/1714903513 learned_addr learned my addr 172.21.15.66:0/1714903513 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:10:25.049 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.049+0000 7f833dd9b640 1 -- 172.21.15.66:0/1714903513 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2847395061 0 0) 0x7f8340142520 con 0x7f834013f710 2024-03-25T18:10:25.050 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.049+0000 7f833dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:10:25.050 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.049+0000 7f833dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:25.050 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.049+0000 7f833dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:25.050 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.049+0000 7f833dd9b640 10 monclient(hunting): my global_id is 4518 2024-03-25T18:10:25.050 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.049+0000 7f833dd9b640 1 -- 172.21.15.66:0/1714903513 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f832c003520 con 0x7f834013f710 2024-03-25T18:10:25.053 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.053+0000 7f833dd9b640 1 -- 172.21.15.66:0/1714903513 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 500957508 0 0) 0x7f83400b28a0 con 0x7f83400a98f0 2024-03-25T18:10:25.053 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.053+0000 7f833dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:10:25.053 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.053+0000 7f833dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:25.054 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.053+0000 7f833dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:25.054 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.053+0000 7f833dd9b640 10 monclient(hunting): my global_id is 4510 2024-03-25T18:10:25.054 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.053+0000 7f833dd9b640 1 -- 172.21.15.66:0/1714903513 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8340142520 con 0x7f83400a98f0 2024-03-25T18:10:25.062 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.061+0000 7f833dd9b640 1 -- 172.21.15.66:0/1714903513 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4251526712 0 0) 0x7f832c003520 con 0x7f834013f710 2024-03-25T18:10:25.062 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.061+0000 7f833dd9b640 1 -- 172.21.15.66:0/1714903513 >> v1:172.21.15.66:6790/0 conn(0x7f83401430c0 legacy=0x7f834014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:25.062 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.061+0000 7f833dd9b640 1 -- 172.21.15.66:0/1714903513 >> v1:172.21.15.193:6789/0 conn(0x7f83400a98f0 legacy=0x7f83400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:25.062 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.061+0000 7f833dd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:10:25.062 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.061+0000 7f833dd9b640 1 monclient: found mon.noname-a 2024-03-25T18:10:25.062 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.061+0000 7f833dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:25.062 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.061+0000 7f833dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.66:6789/0 2024-03-25T18:10:25.062 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.061+0000 7f833dd9b640 1 -- 172.21.15.66:0/1714903513 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83400b44c0 con 0x7f834013f710 2024-03-25T18:10:25.062 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.061+0000 7f833dd9b640 10 monclient: _finish_auth 0 2024-03-25T18:10:25.062 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.061+0000 7f833dd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:10:25.062 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.061+0000 7f833dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:25.063 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.061+0000 7f833dd9b640 1 -- 172.21.15.66:0/1714903513 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8328001690 con 0x7f834013f710 2024-03-25T18:10:25.063 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.061+0000 7f83448e6640 5 monclient: authenticate success, global_id 4518 2024-03-25T18:10:25.063 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.061+0000 7f83448e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:25.063 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.061+0000 7f833dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:25.063 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.061+0000 7f833dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:10:25.063 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.061+0000 7f833dd9b640 10 monclient: dump: 2024-03-25T18:10:25.063 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:25.063 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:25.063 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:25.063 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:25.063 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:25.063 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:25.063 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:25.063 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:25.063 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:25.063 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:25.063 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.061+0000 7f83448e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:25.068 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.067+0000 7f833dd9b640 1 -- 172.21.15.66:0/1714903513 <== mon.0 v1:172.21.15.66:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83280017f0 con 0x7f834013f710 2024-03-25T18:10:25.068 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.067+0000 7f833dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:25.068 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.067+0000 7f833dd9b640 1 -- 172.21.15.66:0/1714903513 <== mon.0 v1:172.21.15.66:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8328001c00 con 0x7f834013f710 2024-03-25T18:10:25.068 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.067+0000 7f833dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:25.068 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.068+0000 7f833dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:25.068 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.068+0000 7f833dd9b640 10 monclient: dump: 2024-03-25T18:10:25.069 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:25.069 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:25.069 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:25.069 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:25.069 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:25.069 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:25.069 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:25.069 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:25.069 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:25.069 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:25.069 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.068+0000 7f83448e6640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:25.069 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.068+0000 7f83448e6640 4 set_mon_vals no callback set 2024-03-25T18:10:25.071 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.068+0000 7f83448e6640 10 monclient: shutdown 2024-03-25T18:10:25.071 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.068+0000 7f83448e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:25.071 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.068+0000 7f83448e6640 1 -- 172.21.15.66:0/1714903513 >> v1:172.21.15.66:6789/0 conn(0x7f834013f710 legacy=0x7f83401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:25.071 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.068+0000 7f83448e6640 1 -- 172.21.15.66:0/1714903513 shutdown_connections 2024-03-25T18:10:25.071 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.068+0000 7f83448e6640 1 -- 172.21.15.66:0/1714903513 >> 172.21.15.66:0/1714903513 conn(0x7f83400b7ea0 msgr2=0x7f83400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:25.071 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.069+0000 7f83448e6640 1 -- 172.21.15.66:0/1714903513 shutdown_connections 2024-03-25T18:10:25.071 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.069+0000 7f83448e6640 1 -- 172.21.15.66:0/1714903513 wait complete. 2024-03-25T18:10:25.071 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.069+0000 7f83448e6640 10 monclient: build_initial_monmap 2024-03-25T18:10:25.071 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.070+0000 7f83448e6640 10 monclient: monmap: 2024-03-25T18:10:25.071 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:10:25.071 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:25.071 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:10:25.072 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:10:25.072 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:25.072 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:10:25.072 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:10:25.072 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:10:25.072 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:10:25.072 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:25.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.070+0000 7f83448e6640 1 librados: starting msgr at 2024-03-25T18:10:25.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.070+0000 7f83448e6640 1 librados: starting objecter 2024-03-25T18:10:25.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.070+0000 7f83448e6640 1 Processor -- start 2024-03-25T18:10:25.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.070+0000 7f83448e6640 1 -- start start 2024-03-25T18:10:25.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.070+0000 7f83448e6640 1 librados: setting wanted keys 2024-03-25T18:10:25.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.070+0000 7f83448e6640 1 librados: calling monclient init 2024-03-25T18:10:25.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.070+0000 7f83448e6640 10 monclient: init 2024-03-25T18:10:25.073 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.071+0000 7f83448e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:25.074 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.071+0000 7f83448e6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:10:25.074 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.071+0000 7f83448e6640 10 monclient(hunting): picked mon.noname-b con 0x7f83400a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:10:25.074 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.071+0000 7f83448e6640 10 monclient(hunting): picked mon.noname-c con 0x7f834013f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:10:25.074 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.071+0000 7f83448e6640 10 monclient(hunting): picked mon.noname-a con 0x7f83401430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:10:25.074 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.071+0000 7f83448e6640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8340137080 con 0x7f83401430c0 2024-03-25T18:10:25.074 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.071+0000 7f83448e6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8340137360 con 0x7f83400a98f0 2024-03-25T18:10:25.074 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.071+0000 7f83448e6640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8340135b10 con 0x7f834013f710 2024-03-25T18:10:25.074 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.071+0000 7f83448e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:25.074 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.071+0000 7f83448e6640 10 monclient(hunting): authenticate will time out at 7374.130555s 2024-03-25T18:10:25.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.078+0000 7f833e59c640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f83400a98f0 0x7f8340137a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:35300/0 (socket says 172.21.15.66:35300) 2024-03-25T18:10:25.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.078+0000 7f833e59c640 1 -- 172.21.15.66:0/4070093215 learned_addr learned my addr 172.21.15.66:0/4070093215 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:10:25.079 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.079+0000 7f8336ffd640 10 client.?.objecter ms_handle_connect 0x7f83400a98f0 2024-03-25T18:10:25.079 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.079+0000 7f8336ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:25.081 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.080+0000 7f8336ffd640 10 client.?.objecter ms_handle_connect 0x7f83401430c0 2024-03-25T18:10:25.081 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.080+0000 7f8336ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:25.088 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.088+0000 7f8336ffd640 1 -- 172.21.15.66:0/4070093215 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1188258881 0 0) 0x7f8340137360 con 0x7f83400a98f0 2024-03-25T18:10:25.088 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.088+0000 7f8336ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:25.088 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.088+0000 7f8336ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:25.088 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.088+0000 7f8336ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:25.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.088+0000 7f8336ffd640 10 monclient(hunting): my global_id is 4513 2024-03-25T18:10:25.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.088+0000 7f8336ffd640 1 -- 172.21.15.66:0/4070093215 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8320004eb0 con 0x7f83400a98f0 2024-03-25T18:10:25.091 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.090+0000 7f8336ffd640 1 -- 172.21.15.66:0/4070093215 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4176369476 0 0) 0x7f8340137080 con 0x7f83401430c0 2024-03-25T18:10:25.091 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.090+0000 7f8336ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:25.091 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.090+0000 7f8336ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:25.091 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.090+0000 7f8336ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:25.091 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.090+0000 7f8336ffd640 10 monclient(hunting): my global_id is 4521 2024-03-25T18:10:25.091 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.090+0000 7f8336ffd640 1 -- 172.21.15.66:0/4070093215 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8340137360 con 0x7f83401430c0 2024-03-25T18:10:25.095 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.094+0000 7f8336ffd640 10 client.?.objecter ms_handle_connect 0x7f834013f710 2024-03-25T18:10:25.095 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.094+0000 7f8336ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:25.099 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.098+0000 7f8336ffd640 1 -- 172.21.15.66:0/4070093215 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 882681312 0 0) 0x7f8320004eb0 con 0x7f83400a98f0 2024-03-25T18:10:25.099 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.098+0000 7f8336ffd640 1 -- 172.21.15.66:0/4070093215 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8340137080 con 0x7f83400a98f0 2024-03-25T18:10:25.099 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.098+0000 7f8336ffd640 1 -- 172.21.15.66:0/4070093215 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f83300019f0 con 0x7f83400a98f0 2024-03-25T18:10:25.099 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.098+0000 7f8336ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:25.099 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.098+0000 7f8336ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:25.099 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.098+0000 7f8336ffd640 10 monclient(hunting): dump: 2024-03-25T18:10:25.099 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:25.099 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:25.099 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:25.099 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:25.099 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:25.099 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:25.100 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:25.100 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:25.100 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:25.100 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:25.100 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.098+0000 7f8336ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:25.109 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.108+0000 7f8336ffd640 1 -- 172.21.15.66:0/4070093215 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3552740530 0 0) 0x7f8340135b10 con 0x7f834013f710 2024-03-25T18:10:25.109 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.109+0000 7f8336ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:25.109 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.109+0000 7f8336ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:25.109 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.109+0000 7f8336ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:25.109 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.109+0000 7f8336ffd640 10 monclient(hunting): my global_id is 4514 2024-03-25T18:10:25.109 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.109+0000 7f8336ffd640 1 -- 172.21.15.66:0/4070093215 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8320004eb0 con 0x7f834013f710 2024-03-25T18:10:25.113 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.112+0000 7f8336ffd640 1 -- 172.21.15.66:0/4070093215 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2313429743 0 0) 0x7f8340137080 con 0x7f83400a98f0 2024-03-25T18:10:25.113 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.112+0000 7f8336ffd640 1 -- 172.21.15.66:0/4070093215 >> v1:172.21.15.66:6790/0 conn(0x7f834013f710 legacy=0x7f83400b6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:25.113 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.112+0000 7f8336ffd640 1 -- 172.21.15.66:0/4070093215 >> v1:172.21.15.66:6789/0 conn(0x7f83401430c0 legacy=0x7f8340130340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:25.113 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.112+0000 7f8336ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:10:25.113 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.112+0000 7f8336ffd640 1 monclient: found mon.b 2024-03-25T18:10:25.113 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.112+0000 7f8336ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:25.113 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.112+0000 7f8336ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:10:25.113 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.112+0000 7f8336ffd640 1 -- 172.21.15.66:0/4070093215 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8340135df0 con 0x7f83400a98f0 2024-03-25T18:10:25.113 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.112+0000 7f8336ffd640 10 monclient: _finish_auth 0 2024-03-25T18:10:25.113 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.112+0000 7f8336ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:10:25.113 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.112+0000 7f8336ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:25.114 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.112+0000 7f83448e6640 5 monclient: authenticate success, global_id 4513 2024-03-25T18:10:25.114 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.112+0000 7f83448e6640 10 monclient: _renew_subs 2024-03-25T18:10:25.114 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.112+0000 7f83448e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:10:25.114 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.112+0000 7f83448e6640 1 -- 172.21.15.66:0/4070093215 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8340136340 con 0x7f83400a98f0 2024-03-25T18:10:25.114 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.112+0000 7f83448e6640 10 client.4513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:25.114 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.112+0000 7f83448e6640 10 monclient: _renew_subs 2024-03-25T18:10:25.114 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.112+0000 7f83448e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:10:25.114 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.112+0000 7f83448e6640 1 -- 172.21.15.66:0/4070093215 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f83401352e0 con 0x7f83400a98f0 2024-03-25T18:10:25.114 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.112+0000 7f83448e6640 1 librados: init done 2024-03-25T18:10:25.115 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.113+0000 7f83448e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:25.115 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.113+0000 7f83448e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:25.115 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.113+0000 7f83448e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:10:25.115 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.113+0000 7f83448e6640 1 -- 172.21.15.66:0/4070093215 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83400627b0 con 0x7f83400a98f0 2024-03-25T18:10:25.383 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.383+0000 7f8336ffd640 1 -- 172.21.15.66:0/4070093215 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83300047e0 con 0x7f83400a98f0 2024-03-25T18:10:25.384 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.383+0000 7f8336ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:25.384 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.383+0000 7f8336ffd640 1 -- 172.21.15.66:0/4070093215 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8330004af0 con 0x7f83400a98f0 2024-03-25T18:10:25.384 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.383+0000 7f8336ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:25.384 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.383+0000 7f8336ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:25.384 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.383+0000 7f8336ffd640 10 monclient: dump: 2024-03-25T18:10:25.384 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:25.384 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:25.384 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:25.384 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:25.384 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:25.384 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:25.384 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:25.384 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:25.384 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:25.385 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:25.385 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.383+0000 7f833d59a640 4 set_mon_vals no callback set 2024-03-25T18:10:25.385 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.384+0000 7f8336ffd640 1 -- 172.21.15.66:0/4070093215 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f833001ab40 con 0x7f83400a98f0 2024-03-25T18:10:25.385 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.384+0000 7f8337fff640 10 monclient: get_auth_request con 0x7f832006de00 auth_method 0 2024-03-25T18:10:25.385 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.385+0000 7f8336ffd640 10 client.4513.objecter ms_handle_connect 0x7f832006de00 2024-03-25T18:10:25.389 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.388+0000 7f8336ffd640 1 -- 172.21.15.66:0/4070093215 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f83300548e0 con 0x7f83400a98f0 2024-03-25T18:10:25.389 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.388+0000 7f8336ffd640 10 client.4513.objecter ms_dispatch 0x7f8340000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:25.389 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.388+0000 7f8336ffd640 3 client.4513.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:25.389 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.388+0000 7f8336ffd640 3 client.4513.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:25.389 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.388+0000 7f8336ffd640 20 client.4513.objecter dump_active .. 0 homeless 2024-03-25T18:10:25.389 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.389+0000 7f8336ffd640 1 -- 172.21.15.66:0/4070093215 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8330050220 con 0x7f83400a98f0 2024-03-25T18:10:25.389 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.389+0000 7f8336ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:25.389 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.389+0000 7f8336ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:25.621 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.620+0000 7f83448e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:25.621 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.620+0000 7f83448e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:25.621 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.620+0000 7f83448e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:10:25.621 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.620+0000 7f83448e6640 1 -- 172.21.15.66:0/4070093215 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f834014f110 con 0x7f83400a98f0 2024-03-25T18:10:25.652 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.651+0000 7f8336ffd640 1 -- 172.21.15.66:0/4070093215 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f8330052320 con 0x7f83400a98f0 2024-03-25T18:10:25.652 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.651+0000 7f8336ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:25.652 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.651+0000 7f8336ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:25.652 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:10:25.652 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:25.655 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.653+0000 7f83448e6640 10 librados: watch_flush enter 2024-03-25T18:10:25.655 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.653+0000 7f83448e6640 10 librados: watch_flush exit 2024-03-25T18:10:25.655 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.653+0000 7f83448e6640 20 client.4513.objecter shutdown clearing up homeless session... 2024-03-25T18:10:25.655 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.653+0000 7f83448e6640 10 client.4513.objecter successfully canceled tick 2024-03-25T18:10:25.655 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.654+0000 7f83448e6640 1 -- 172.21.15.66:0/4070093215 >> v1:172.21.15.193:6800/2275676803 conn(0x7f832006de00 legacy=0x7f83200702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:25.655 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.654+0000 7f83448e6640 10 monclient: shutdown 2024-03-25T18:10:25.655 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.654+0000 7f83448e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:25.655 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.654+0000 7f83448e6640 1 -- 172.21.15.66:0/4070093215 >> v1:172.21.15.193:6789/0 conn(0x7f83400a98f0 legacy=0x7f8340137a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:25.655 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.654+0000 7f83448e6640 1 -- 172.21.15.66:0/4070093215 shutdown_connections 2024-03-25T18:10:25.655 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.654+0000 7f83448e6640 1 -- 172.21.15.66:0/4070093215 >> 172.21.15.66:0/4070093215 conn(0x7f83400b7ea0 msgr2=0x7f8340074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:25.655 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.654+0000 7f83448e6640 1 -- 172.21.15.66:0/4070093215 shutdown_connections 2024-03-25T18:10:25.656 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.654+0000 7f83448e6640 1 -- 172.21.15.66:0/4070093215 wait complete. 2024-03-25T18:10:25.656 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:25.654+0000 7f83448e6640 1 librados: shutdown 2024-03-25T18:10:25.665 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:29.689 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:07:00.669 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:29.740 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:07:00.720 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:29.770 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:07:00.751 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:29.772 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:07:00.731 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:29.807 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:07:00.765 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:29.812 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:07:00.793 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:29.817 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:07:00.797 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:29.844 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:07:00.824 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:29.845 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:07:00.804 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:32.471 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:32.594 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.592+0000 7fdaa92e2640 10 monclient: get_monmap_and_config 2024-03-25T18:10:32.594 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.592+0000 7fdaa92e2640 10 monclient: build_initial_monmap 2024-03-25T18:10:32.594 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.592+0000 7fdaa92e2640 10 monclient: monmap: 2024-03-25T18:10:32.594 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:10:32.594 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:32.595 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:10:32.594435+0000 2024-03-25T18:10:32.595 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:10:32.594435+0000 2024-03-25T18:10:32.595 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:32.595 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:32.595 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:10:32.595 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:10:32.595 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:10:32.595 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:32.596 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.593+0000 7fdaa92e2640 1 Processor -- start 2024-03-25T18:10:32.596 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.593+0000 7fdaa92e2640 1 -- start start 2024-03-25T18:10:32.596 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.593+0000 7fdaa92e2640 10 monclient: init 2024-03-25T18:10:32.597 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.594+0000 7fdaa92e2640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:32.597 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.594+0000 7fdaa92e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:10:32.597 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.594+0000 7fdaa92e2640 10 monclient(hunting): picked mon.noname-b con 0x7fdaa40a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:10:32.597 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.594+0000 7fdaa92e2640 10 monclient(hunting): picked mon.noname-a con 0x7fdaa413f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:10:32.597 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.594+0000 7fdaa92e2640 10 monclient(hunting): picked mon.noname-c con 0x7fdaa41430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:10:32.597 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.594+0000 7fdaa92e2640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa4142520 con 0x7fdaa413f710 2024-03-25T18:10:32.597 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.594+0000 7fdaa92e2640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa40b28a0 con 0x7fdaa41430c0 2024-03-25T18:10:32.597 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.594+0000 7fdaa92e2640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa40b2a30 con 0x7fdaa40a98f0 2024-03-25T18:10:32.597 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.594+0000 7fdaa92e2640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:32.597 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.594+0000 7fdaa92e2640 10 monclient(hunting): authenticate will time out at 7381.654846s 2024-03-25T18:10:32.601 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.599+0000 7fdaa37fe640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fdaa41430c0 0x7fdaa414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:39824/0 (socket says 172.21.15.66:39824) 2024-03-25T18:10:32.601 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.599+0000 7fdaa37fe640 1 -- 172.21.15.66:0/2075042438 learned_addr learned my addr 172.21.15.66:0/2075042438 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:10:32.610 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.609+0000 7fdaa27fc640 1 -- 172.21.15.66:0/2075042438 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2357104216 0 0) 0x7fdaa40b28a0 con 0x7fdaa41430c0 2024-03-25T18:10:32.611 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.609+0000 7fdaa27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:10:32.611 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.609+0000 7fdaa27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:32.611 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.609+0000 7fdaa27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:32.611 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.609+0000 7fdaa27fc640 10 monclient(hunting): my global_id is 4516 2024-03-25T18:10:32.611 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.609+0000 7fdaa27fc640 1 -- 172.21.15.66:0/2075042438 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda90003520 con 0x7fdaa41430c0 2024-03-25T18:10:32.618 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.617+0000 7fdaa27fc640 1 -- 172.21.15.66:0/2075042438 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3186656306 0 0) 0x7fda90003520 con 0x7fdaa41430c0 2024-03-25T18:10:32.618 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.617+0000 7fdaa27fc640 1 -- 172.21.15.66:0/2075042438 >> v1:172.21.15.66:6790/0 conn(0x7fdaa40a98f0 legacy=0x7fdaa40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:32.618 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.617+0000 7fdaa27fc640 1 -- 172.21.15.66:0/2075042438 >> v1:172.21.15.66:6789/0 conn(0x7fdaa413f710 legacy=0x7fdaa41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:32.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.617+0000 7fdaa27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:10:32.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.617+0000 7fdaa27fc640 1 monclient: found mon.noname-c 2024-03-25T18:10:32.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.617+0000 7fdaa27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:32.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.617+0000 7fdaa27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:10:32.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.617+0000 7fdaa27fc640 1 -- 172.21.15.66:0/2075042438 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa40b44c0 con 0x7fdaa41430c0 2024-03-25T18:10:32.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.617+0000 7fdaa27fc640 10 monclient: _finish_auth 0 2024-03-25T18:10:32.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.617+0000 7fdaa27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:10:32.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.617+0000 7fdaa27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:32.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.617+0000 7fdaa27fc640 1 -- 172.21.15.66:0/2075042438 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fda9c001a10 con 0x7fdaa41430c0 2024-03-25T18:10:32.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.617+0000 7fdaa27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:32.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.617+0000 7fdaa27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:32.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.617+0000 7fdaa27fc640 10 monclient: dump: 2024-03-25T18:10:32.619 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:32.619 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:32.619 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:32.620 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:32.620 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:32.620 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:32.620 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:32.620 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:32.620 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:32.620 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:32.620 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.617+0000 7fdaa92e2640 5 monclient: authenticate success, global_id 4516 2024-03-25T18:10:32.620 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.617+0000 7fdaa92e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:32.644 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.643+0000 7fdaa27fc640 1 -- 172.21.15.66:0/2075042438 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda9c001b70 con 0x7fdaa41430c0 2024-03-25T18:10:32.644 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.643+0000 7fdaa27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:32.644 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.643+0000 7fdaa27fc640 1 -- 172.21.15.66:0/2075042438 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fda9c001f80 con 0x7fdaa41430c0 2024-03-25T18:10:32.644 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.643+0000 7fdaa92e2640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:32.644 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.643+0000 7fdaa92e2640 4 set_mon_vals no callback set 2024-03-25T18:10:32.645 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.643+0000 7fdaa27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:32.645 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.643+0000 7fdaa27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:32.645 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.643+0000 7fdaa27fc640 10 monclient: dump: 2024-03-25T18:10:32.645 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:32.645 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:32.645 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:32.645 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:32.645 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:32.645 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:32.645 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:32.645 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:32.645 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:32.645 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:32.647 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.643+0000 7fdaa92e2640 10 monclient: shutdown 2024-03-25T18:10:32.647 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.643+0000 7fdaa92e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:32.647 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.643+0000 7fdaa92e2640 1 -- 172.21.15.66:0/2075042438 >> v1:172.21.15.193:6789/0 conn(0x7fdaa41430c0 legacy=0x7fdaa414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:32.647 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.644+0000 7fdaa92e2640 1 -- 172.21.15.66:0/2075042438 shutdown_connections 2024-03-25T18:10:32.647 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.644+0000 7fdaa92e2640 1 -- 172.21.15.66:0/2075042438 >> 172.21.15.66:0/2075042438 conn(0x7fdaa40b7ea0 msgr2=0x7fdaa40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:32.647 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.644+0000 7fdaa92e2640 1 -- 172.21.15.66:0/2075042438 shutdown_connections 2024-03-25T18:10:32.647 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.644+0000 7fdaa92e2640 1 -- 172.21.15.66:0/2075042438 wait complete. 2024-03-25T18:10:32.647 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.644+0000 7fdaa92e2640 10 monclient: build_initial_monmap 2024-03-25T18:10:32.647 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.644+0000 7fdaa92e2640 10 monclient: monmap: 2024-03-25T18:10:32.647 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:10:32.647 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:32.647 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:10:32.647 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:10:32.648 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:32.648 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:10:32.648 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:10:32.648 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:10:32.648 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:10:32.648 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:32.648 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.645+0000 7fdaa92e2640 1 librados: starting msgr at 2024-03-25T18:10:32.648 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.645+0000 7fdaa92e2640 1 librados: starting objecter 2024-03-25T18:10:32.648 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.645+0000 7fdaa92e2640 1 Processor -- start 2024-03-25T18:10:32.648 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.645+0000 7fdaa92e2640 1 -- start start 2024-03-25T18:10:32.648 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.645+0000 7fdaa92e2640 1 librados: setting wanted keys 2024-03-25T18:10:32.648 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.645+0000 7fdaa92e2640 1 librados: calling monclient init 2024-03-25T18:10:32.648 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.645+0000 7fdaa92e2640 10 monclient: init 2024-03-25T18:10:32.649 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.646+0000 7fdaa92e2640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:32.649 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.646+0000 7fdaa92e2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:10:32.650 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.646+0000 7fdaa92e2640 10 monclient(hunting): picked mon.noname-b con 0x7fdaa40a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:10:32.650 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.646+0000 7fdaa92e2640 10 monclient(hunting): picked mon.noname-a con 0x7fdaa413f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:10:32.650 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.646+0000 7fdaa92e2640 10 monclient(hunting): picked mon.noname-c con 0x7fdaa41430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:10:32.650 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.646+0000 7fdaa92e2640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa4137ab0 con 0x7fdaa413f710 2024-03-25T18:10:32.650 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.646+0000 7fdaa92e2640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa4138bd0 con 0x7fdaa40a98f0 2024-03-25T18:10:32.650 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.646+0000 7fdaa92e2640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa4138e40 con 0x7fdaa41430c0 2024-03-25T18:10:32.650 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.646+0000 7fdaa92e2640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:32.650 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.646+0000 7fdaa92e2640 10 monclient(hunting): authenticate will time out at 7381.706324s 2024-03-25T18:10:32.653 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.651+0000 7fdaa37fe640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7fdaa41430c0 0x7fdaa41e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:48664/0 (socket says 172.21.15.66:48664) 2024-03-25T18:10:32.653 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.652+0000 7fdaa37fe640 1 -- 172.21.15.66:0/3530720826 learned_addr learned my addr 172.21.15.66:0/3530720826 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:10:32.657 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.655+0000 7fda9b7fe640 10 client.?.objecter ms_handle_connect 0x7fdaa41430c0 2024-03-25T18:10:32.657 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.655+0000 7fda9b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:32.660 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.659+0000 7fda9b7fe640 1 -- 172.21.15.66:0/3530720826 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2353504879 0 0) 0x7fdaa4138e40 con 0x7fdaa41430c0 2024-03-25T18:10:32.660 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.659+0000 7fda9b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:10:32.660 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.659+0000 7fda9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:32.660 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.659+0000 7fda9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:32.660 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.659+0000 7fda9b7fe640 10 monclient(hunting): my global_id is 4520 2024-03-25T18:10:32.660 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.659+0000 7fda9b7fe640 1 -- 172.21.15.66:0/3530720826 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda84004eb0 con 0x7fdaa41430c0 2024-03-25T18:10:32.661 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.659+0000 7fda9b7fe640 10 client.?.objecter ms_handle_connect 0x7fdaa40a98f0 2024-03-25T18:10:32.661 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.659+0000 7fda9b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:32.667 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.666+0000 7fda9b7fe640 10 client.?.objecter ms_handle_connect 0x7fdaa413f710 2024-03-25T18:10:32.667 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.666+0000 7fda9b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:32.668 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.667+0000 7fda9b7fe640 1 -- 172.21.15.66:0/3530720826 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1429390878 0 0) 0x7fda84004eb0 con 0x7fdaa41430c0 2024-03-25T18:10:32.668 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.667+0000 7fda9b7fe640 1 -- 172.21.15.66:0/3530720826 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdaa4138e40 con 0x7fdaa41430c0 2024-03-25T18:10:32.669 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.667+0000 7fda9b7fe640 1 -- 172.21.15.66:0/3530720826 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fda9c001f40 con 0x7fdaa41430c0 2024-03-25T18:10:32.669 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.667+0000 7fda9b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:32.669 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.667+0000 7fda9b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:32.669 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.667+0000 7fda9b7fe640 10 monclient(hunting): dump: 2024-03-25T18:10:32.669 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:32.669 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:32.669 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:32.669 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:32.669 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:32.669 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:32.669 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:32.669 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:32.669 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:32.669 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:32.669 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.667+0000 7fda9b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:32.674 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.672+0000 7fda9b7fe640 1 -- 172.21.15.66:0/3530720826 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 247393077 0 0) 0x7fdaa4138bd0 con 0x7fdaa40a98f0 2024-03-25T18:10:32.674 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.672+0000 7fda9b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:10:32.674 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.672+0000 7fda9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:32.674 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.672+0000 7fda9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:32.674 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.672+0000 7fda9b7fe640 10 monclient(hunting): my global_id is 4519 2024-03-25T18:10:32.674 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.672+0000 7fda9b7fe640 1 -- 172.21.15.66:0/3530720826 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda84004eb0 con 0x7fdaa40a98f0 2024-03-25T18:10:32.678 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.675+0000 7fda9b7fe640 1 -- 172.21.15.66:0/3530720826 <== mon.2 v1:172.21.15.66:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 435330384 0 0) 0x7fdaa4138e40 con 0x7fdaa41430c0 2024-03-25T18:10:32.678 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.675+0000 7fda9b7fe640 1 -- 172.21.15.66:0/3530720826 >> v1:172.21.15.193:6789/0 conn(0x7fdaa40a98f0 legacy=0x7fdaa4137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:32.678 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.675+0000 7fda9b7fe640 1 -- 172.21.15.66:0/3530720826 >> v1:172.21.15.66:6789/0 conn(0x7fdaa413f710 legacy=0x7fdaa41df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:32.678 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.675+0000 7fda9b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:10:32.678 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.675+0000 7fda9b7fe640 1 monclient: found mon.c 2024-03-25T18:10:32.678 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.676+0000 7fda9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:32.678 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.676+0000 7fda9b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:10:32.678 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.676+0000 7fda9b7fe640 1 -- 172.21.15.66:0/3530720826 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa4139120 con 0x7fdaa41430c0 2024-03-25T18:10:32.678 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.676+0000 7fda9b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:10:32.678 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.676+0000 7fda9b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:10:32.678 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.676+0000 7fda9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:32.678 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.676+0000 7fdaa92e2640 5 monclient: authenticate success, global_id 4520 2024-03-25T18:10:32.678 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.676+0000 7fdaa92e2640 10 monclient: _renew_subs 2024-03-25T18:10:32.679 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.676+0000 7fdaa92e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:10:32.679 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.676+0000 7fdaa92e2640 1 -- 172.21.15.66:0/3530720826 --> v1:172.21.15.66:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaa4139400 con 0x7fdaa41430c0 2024-03-25T18:10:32.679 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.676+0000 7fdaa92e2640 10 client.4520.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:32.679 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.676+0000 7fdaa92e2640 10 monclient: _renew_subs 2024-03-25T18:10:32.679 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.676+0000 7fdaa92e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:10:32.679 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.676+0000 7fdaa92e2640 1 -- 172.21.15.66:0/3530720826 --> v1:172.21.15.66:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaa40b7880 con 0x7fdaa41430c0 2024-03-25T18:10:32.679 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.676+0000 7fdaa92e2640 1 librados: init done 2024-03-25T18:10:32.679 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.677+0000 7fdaa92e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:32.680 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.677+0000 7fdaa92e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:32.680 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.677+0000 7fdaa92e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:10:32.680 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.677+0000 7fdaa92e2640 1 -- 172.21.15.66:0/3530720826 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaa4137810 con 0x7fdaa41430c0 2024-03-25T18:10:32.986 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.985+0000 7fda9b7fe640 1 -- 172.21.15.66:0/3530720826 <== mon.2 v1:172.21.15.66:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda9c004e10 con 0x7fdaa41430c0 2024-03-25T18:10:32.987 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.985+0000 7fda9b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:32.987 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.985+0000 7fda9b7fe640 1 -- 172.21.15.66:0/3530720826 <== mon.2 v1:172.21.15.66:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fda9c005120 con 0x7fdaa41430c0 2024-03-25T18:10:32.987 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.985+0000 7fda9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:32.987 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.985+0000 7fda9b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:32.987 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.985+0000 7fda9b7fe640 10 monclient: dump: 2024-03-25T18:10:32.987 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:32.987 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:32.987 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:32.987 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:32.987 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:32.987 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:32.987 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:32.987 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:32.988 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:32.988 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:32.988 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.985+0000 7fdaa1ffb640 4 set_mon_vals no callback set 2024-03-25T18:10:32.988 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.986+0000 7fda9b7fe640 1 -- 172.21.15.66:0/3530720826 <== mon.2 v1:172.21.15.66:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7fda9c01b0f0 con 0x7fdaa41430c0 2024-03-25T18:10:32.988 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.986+0000 7fdaa2ffd640 10 monclient: get_auth_request con 0x7fda8406dd40 auth_method 0 2024-03-25T18:10:32.988 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.987+0000 7fda9b7fe640 10 client.4520.objecter ms_handle_connect 0x7fda8406dd40 2024-03-25T18:10:32.995 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.993+0000 7fda9b7fe640 1 -- 172.21.15.66:0/3530720826 <== mon.2 v1:172.21.15.66:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7fda9c054bc0 con 0x7fdaa41430c0 2024-03-25T18:10:32.995 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.993+0000 7fda9b7fe640 10 client.4520.objecter ms_dispatch 0x7fdaa4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:32.995 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.993+0000 7fda9b7fe640 3 client.4520.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:32.995 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.993+0000 7fda9b7fe640 3 client.4520.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:32.995 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.993+0000 7fda9b7fe640 20 client.4520.objecter dump_active .. 0 homeless 2024-03-25T18:10:32.995 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.993+0000 7fda9b7fe640 1 -- 172.21.15.66:0/3530720826 <== mon.2 v1:172.21.15.66:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fda9c0506d0 con 0x7fdaa41430c0 2024-03-25T18:10:32.995 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.993+0000 7fda9b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:32.995 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:32.993+0000 7fda9b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:33.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:33.251+0000 7fdaa92e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:33.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:33.251+0000 7fdaa92e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:33.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:33.251+0000 7fdaa92e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:10:33.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:33.251+0000 7fdaa92e2640 1 -- 172.21.15.66:0/3530720826 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdaa40627b0 con 0x7fdaa41430c0 2024-03-25T18:10:33.292 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:33.290+0000 7fda9b7fe640 1 -- 172.21.15.66:0/3530720826 <== mon.2 v1:172.21.15.66:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7fda9c052600 con 0x7fdaa41430c0 2024-03-25T18:10:33.292 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:33.290+0000 7fda9b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:33.292 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:33.290+0000 7fda9b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:33.292 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:10:33.292 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:33.295 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:33.293+0000 7fdaa92e2640 10 librados: watch_flush enter 2024-03-25T18:10:33.295 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:33.293+0000 7fdaa92e2640 10 librados: watch_flush exit 2024-03-25T18:10:33.295 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:33.293+0000 7fdaa92e2640 20 client.4520.objecter shutdown clearing up homeless session... 2024-03-25T18:10:33.295 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:33.293+0000 7fdaa92e2640 10 client.4520.objecter successfully canceled tick 2024-03-25T18:10:33.295 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:33.293+0000 7fdaa92e2640 1 -- 172.21.15.66:0/3530720826 >> v1:172.21.15.193:6800/2275676803 conn(0x7fda8406dd40 legacy=0x7fda84070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:33.295 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:33.293+0000 7fdaa92e2640 10 monclient: shutdown 2024-03-25T18:10:33.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:33.293+0000 7fdaa92e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:33.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:33.293+0000 7fdaa92e2640 1 -- 172.21.15.66:0/3530720826 >> v1:172.21.15.66:6790/0 conn(0x7fdaa41430c0 legacy=0x7fdaa41e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:33.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:33.293+0000 7fdaa92e2640 1 -- 172.21.15.66:0/3530720826 shutdown_connections 2024-03-25T18:10:33.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:33.293+0000 7fdaa92e2640 1 -- 172.21.15.66:0/3530720826 >> 172.21.15.66:0/3530720826 conn(0x7fdaa40b7ea0 msgr2=0x7fdaa4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:33.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:33.293+0000 7fdaa92e2640 1 -- 172.21.15.66:0/3530720826 shutdown_connections 2024-03-25T18:10:33.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:33.293+0000 7fdaa92e2640 1 -- 172.21.15.66:0/3530720826 wait complete. 2024-03-25T18:10:33.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:33.293+0000 7fdaa92e2640 1 librados: shutdown 2024-03-25T18:10:33.308 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:40.114 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:40.232 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.231+0000 7f634a4ed640 10 monclient: get_monmap_and_config 2024-03-25T18:10:40.232 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.231+0000 7f634a4ed640 10 monclient: build_initial_monmap 2024-03-25T18:10:40.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.231+0000 7f634a4ed640 10 monclient: monmap: 2024-03-25T18:10:40.233 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:10:40.233 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:40.233 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:10:40.232681+0000 2024-03-25T18:10:40.233 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:10:40.232681+0000 2024-03-25T18:10:40.233 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:40.233 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:40.233 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:10:40.233 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:10:40.233 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:10:40.233 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:40.234 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.231+0000 7f634a4ed640 1 Processor -- start 2024-03-25T18:10:40.235 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.232+0000 7f634a4ed640 1 -- start start 2024-03-25T18:10:40.235 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.232+0000 7f634a4ed640 10 monclient: init 2024-03-25T18:10:40.235 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.232+0000 7f634a4ed640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:40.235 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.232+0000 7f634a4ed640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:10:40.235 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.232+0000 7f634a4ed640 10 monclient(hunting): picked mon.noname-b con 0x7f63440b4d60 addr v1:172.21.15.66:6790/0 2024-03-25T18:10:40.235 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.232+0000 7f634a4ed640 10 monclient(hunting): picked mon.noname-c con 0x7f63440b1730 addr v1:172.21.15.193:6789/0 2024-03-25T18:10:40.235 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.232+0000 7f634a4ed640 10 monclient(hunting): picked mon.noname-a con 0x7f634414be90 addr v1:172.21.15.66:6789/0 2024-03-25T18:10:40.235 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.232+0000 7f634a4ed640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63440af940 con 0x7f634414be90 2024-03-25T18:10:40.235 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.232+0000 7f634a4ed640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63440b0a70 con 0x7f63440b1730 2024-03-25T18:10:40.235 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.232+0000 7f634a4ed640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63440b0d50 con 0x7f63440b4d60 2024-03-25T18:10:40.235 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.232+0000 7f634a4ed640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:40.235 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.232+0000 7f634a4ed640 10 monclient(hunting): authenticate will time out at 7389.292747s 2024-03-25T18:10:40.244 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.242+0000 7f6343fff640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f63440b4d60 0x7f634413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:50990/0 (socket says 172.21.15.66:50990) 2024-03-25T18:10:40.244 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.242+0000 7f6343fff640 1 -- 172.21.15.66:0/1417875921 learned_addr learned my addr 172.21.15.66:0/1417875921 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:10:40.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.252+0000 7f6342ffd640 1 -- 172.21.15.66:0/1417875921 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 621101504 0 0) 0x7f63440b0d50 con 0x7f63440b4d60 2024-03-25T18:10:40.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.252+0000 7f6342ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:40.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.252+0000 7f6342ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:40.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.252+0000 7f6342ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:40.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.252+0000 7f6342ffd640 10 monclient(hunting): my global_id is 4523 2024-03-25T18:10:40.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.252+0000 7f6342ffd640 1 -- 172.21.15.66:0/1417875921 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f632c003520 con 0x7f63440b4d60 2024-03-25T18:10:40.266 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.263+0000 7f6342ffd640 1 -- 172.21.15.66:0/1417875921 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2546468195 0 0) 0x7f632c003520 con 0x7f63440b4d60 2024-03-25T18:10:40.266 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.264+0000 7f6342ffd640 1 -- 172.21.15.66:0/1417875921 >> v1:172.21.15.193:6789/0 conn(0x7f63440b1730 legacy=0x7f63440b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:40.266 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.264+0000 7f6342ffd640 1 -- 172.21.15.66:0/1417875921 >> v1:172.21.15.66:6789/0 conn(0x7f634414be90 legacy=0x7f634414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:40.266 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.264+0000 7f6342ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:10:40.266 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.264+0000 7f6342ffd640 1 monclient: found mon.noname-b 2024-03-25T18:10:40.266 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.264+0000 7f6342ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:40.266 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.264+0000 7f6342ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.66:6790/0 2024-03-25T18:10:40.266 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.264+0000 7f6342ffd640 1 -- 172.21.15.66:0/1417875921 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63440b1030 con 0x7f63440b4d60 2024-03-25T18:10:40.266 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.264+0000 7f6342ffd640 10 monclient: _finish_auth 0 2024-03-25T18:10:40.266 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.264+0000 7f6342ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:10:40.267 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.264+0000 7f6342ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:40.267 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.264+0000 7f634a4ed640 5 monclient: authenticate success, global_id 4523 2024-03-25T18:10:40.267 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.264+0000 7f634a4ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:40.267 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.264+0000 7f6342ffd640 1 -- 172.21.15.66:0/1417875921 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f6334001690 con 0x7f63440b4d60 2024-03-25T18:10:40.267 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.264+0000 7f6342ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:40.267 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.264+0000 7f6342ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:40.267 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.264+0000 7f6342ffd640 10 monclient: dump: 2024-03-25T18:10:40.267 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:40.267 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:40.267 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:40.267 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:40.267 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:40.267 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:40.267 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:40.267 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:40.267 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:40.268 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:40.268 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.264+0000 7f634a4ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:40.292 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.291+0000 7f6342ffd640 1 -- 172.21.15.66:0/1417875921 <== mon.2 v1:172.21.15.66:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6334001920 con 0x7f63440b4d60 2024-03-25T18:10:40.292 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.291+0000 7f6342ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:40.292 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.291+0000 7f6342ffd640 1 -- 172.21.15.66:0/1417875921 <== mon.2 v1:172.21.15.66:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f6334001c30 con 0x7f63440b4d60 2024-03-25T18:10:40.292 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.291+0000 7f6342ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:40.292 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.291+0000 7f6342ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:40.293 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.291+0000 7f6342ffd640 10 monclient: dump: 2024-03-25T18:10:40.293 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:40.293 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:40.293 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:40.293 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:40.293 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:40.293 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:40.293 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:40.293 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:40.293 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:40.293 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:40.293 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.291+0000 7f634a4ed640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:40.293 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.291+0000 7f634a4ed640 4 set_mon_vals no callback set 2024-03-25T18:10:40.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.291+0000 7f634a4ed640 10 monclient: shutdown 2024-03-25T18:10:40.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.291+0000 7f634a4ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:40.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.291+0000 7f634a4ed640 1 -- 172.21.15.66:0/1417875921 >> v1:172.21.15.66:6790/0 conn(0x7f63440b4d60 legacy=0x7f634413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:40.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.292+0000 7f634a4ed640 1 -- 172.21.15.66:0/1417875921 shutdown_connections 2024-03-25T18:10:40.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.292+0000 7f634a4ed640 1 -- 172.21.15.66:0/1417875921 >> 172.21.15.66:0/1417875921 conn(0x7f63440617a0 msgr2=0x7f6344061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:40.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.292+0000 7f634a4ed640 1 -- 172.21.15.66:0/1417875921 shutdown_connections 2024-03-25T18:10:40.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.292+0000 7f634a4ed640 1 -- 172.21.15.66:0/1417875921 wait complete. 2024-03-25T18:10:40.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.292+0000 7f634a4ed640 10 monclient: build_initial_monmap 2024-03-25T18:10:40.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.292+0000 7f634a4ed640 10 monclient: monmap: 2024-03-25T18:10:40.296 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:10:40.296 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:40.297 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:10:40.297 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:10:40.297 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:40.297 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:10:40.297 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:10:40.297 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:10:40.297 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:10:40.297 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:40.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.293+0000 7f634a4ed640 1 librados: starting msgr at 2024-03-25T18:10:40.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.293+0000 7f634a4ed640 1 librados: starting objecter 2024-03-25T18:10:40.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.293+0000 7f634a4ed640 1 Processor -- start 2024-03-25T18:10:40.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.293+0000 7f634a4ed640 1 -- start start 2024-03-25T18:10:40.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.293+0000 7f634a4ed640 1 librados: setting wanted keys 2024-03-25T18:10:40.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.293+0000 7f634a4ed640 1 librados: calling monclient init 2024-03-25T18:10:40.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.293+0000 7f634a4ed640 10 monclient: init 2024-03-25T18:10:40.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.294+0000 7f634a4ed640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:40.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.294+0000 7f634a4ed640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:10:40.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.294+0000 7f634a4ed640 10 monclient(hunting): picked mon.noname-b con 0x7f63440b1730 addr v1:172.21.15.193:6789/0 2024-03-25T18:10:40.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.294+0000 7f634a4ed640 10 monclient(hunting): picked mon.noname-a con 0x7f63440b4d60 addr v1:172.21.15.66:6789/0 2024-03-25T18:10:40.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.294+0000 7f634a4ed640 10 monclient(hunting): picked mon.noname-c con 0x7f634414be90 addr v1:172.21.15.66:6790/0 2024-03-25T18:10:40.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.294+0000 7f634a4ed640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63441d7ba0 con 0x7f63440b4d60 2024-03-25T18:10:40.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.294+0000 7f634a4ed640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63441d8740 con 0x7f63440b1730 2024-03-25T18:10:40.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.294+0000 7f634a4ed640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63441d7830 con 0x7f634414be90 2024-03-25T18:10:40.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.294+0000 7f634a4ed640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:40.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.294+0000 7f634a4ed640 10 monclient(hunting): authenticate will time out at 7389.354562s 2024-03-25T18:10:40.304 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.303+0000 7f63437fe640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f63440b4d60 0x7f6344135f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:51930/0 (socket says 172.21.15.66:51930) 2024-03-25T18:10:40.304 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.303+0000 7f63437fe640 1 -- 172.21.15.66:0/3309937963 learned_addr learned my addr 172.21.15.66:0/3309937963 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:10:40.306 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.304+0000 7f6340ff9640 10 client.?.objecter ms_handle_connect 0x7f63440b4d60 2024-03-25T18:10:40.306 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.304+0000 7f6340ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:40.308 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.307+0000 7f6340ff9640 10 client.?.objecter ms_handle_connect 0x7f63440b1730 2024-03-25T18:10:40.309 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.307+0000 7f6340ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:40.317 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.316+0000 7f6340ff9640 1 -- 172.21.15.66:0/3309937963 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3176273764 0 0) 0x7f63441d7ba0 con 0x7f63440b4d60 2024-03-25T18:10:40.317 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.316+0000 7f6340ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:10:40.318 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.316+0000 7f6340ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:40.318 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.316+0000 7f6340ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:40.318 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.316+0000 7f6340ff9640 10 monclient(hunting): my global_id is 4530 2024-03-25T18:10:40.318 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.316+0000 7f6340ff9640 1 -- 172.21.15.66:0/3309937963 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6330004eb0 con 0x7f63440b4d60 2024-03-25T18:10:40.319 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.317+0000 7f6340ff9640 10 client.?.objecter ms_handle_connect 0x7f634414be90 2024-03-25T18:10:40.319 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.317+0000 7f6340ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:40.323 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.322+0000 7f63437fe640 1 -- 172.21.15.66:0/3309937963 >> v1:172.21.15.66:6789/0 conn(0x7f63440b4d60 legacy=0x7f6344135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T18:10:40.323 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.322+0000 7f63437fe640 1 -- 172.21.15.66:0/3309937963 >> v1:172.21.15.66:6789/0 conn(0x7f63440b4d60 legacy=0x7f6344135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:10:40.323 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.322+0000 7f63437fe640 1 --1- 172.21.15.66:0/3309937963 >> v1:172.21.15.66:6789/0 conn(0x7f63440b4d60 0x7f6344135f70 :-1 s=OPENED pgs=48640 cs=1 l=1).handle_message read tag failed 2024-03-25T18:10:40.323 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.322+0000 7f63437fe640 1 --1- 172.21.15.66:0/3309937963 >> v1:172.21.15.66:6789/0 conn(0x7f63440b4d60 0x7f6344135f70 :-1 s=OPENED pgs=48640 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:10:40.323 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.322+0000 7f6340ff9640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.66:6789/0 2024-03-25T18:10:40.438 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.436+0000 7f6340ff9640 1 -- 172.21.15.66:0/3309937963 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013517819 0 0) 0x7f63441d8740 con 0x7f63440b1730 2024-03-25T18:10:40.438 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.437+0000 7f6340ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:10:40.438 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.437+0000 7f6340ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:40.438 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.437+0000 7f6340ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:40.439 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.437+0000 7f6340ff9640 10 monclient(hunting): my global_id is 4525 2024-03-25T18:10:40.439 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.437+0000 7f6340ff9640 1 -- 172.21.15.66:0/3309937963 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63441d7ba0 con 0x7f63440b1730 2024-03-25T18:10:40.449 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.448+0000 7f6340ff9640 1 -- 172.21.15.66:0/3309937963 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1214464198 0 0) 0x7f63441d7ba0 con 0x7f63440b1730 2024-03-25T18:10:40.449 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.448+0000 7f6340ff9640 1 -- 172.21.15.66:0/3309937963 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63441d8740 con 0x7f63440b1730 2024-03-25T18:10:40.449 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.448+0000 7f6340ff9640 1 -- 172.21.15.66:0/3309937963 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f63340014b0 con 0x7f63440b1730 2024-03-25T18:10:40.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.448+0000 7f6340ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:40.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.448+0000 7f6340ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:40.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.448+0000 7f6340ff9640 10 monclient(hunting): dump: 2024-03-25T18:10:40.450 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:40.450 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:40.450 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:40.450 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:40.450 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:40.450 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:40.450 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:40.450 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:40.450 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:40.450 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:40.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.448+0000 7f6340ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:40.464 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.462+0000 7f6340ff9640 1 -- 172.21.15.66:0/3309937963 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2048568777 0 0) 0x7f63441d8740 con 0x7f63440b1730 2024-03-25T18:10:40.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.462+0000 7f6340ff9640 1 -- 172.21.15.66:0/3309937963 >> v1:172.21.15.66:6790/0 conn(0x7f634414be90 legacy=0x7f63441d8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:40.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.462+0000 7f6340ff9640 1 -- 172.21.15.66:0/3309937963 >> v1:172.21.15.66:6789/0 conn(0x7f63440b4d60 legacy=0x7f6344135f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:10:40.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.462+0000 7f6340ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:10:40.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.462+0000 7f6340ff9640 1 monclient: found mon.b 2024-03-25T18:10:40.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.462+0000 7f6340ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:40.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.462+0000 7f6340ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:10:40.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.462+0000 7f6340ff9640 1 -- 172.21.15.66:0/3309937963 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6344143900 con 0x7f63440b1730 2024-03-25T18:10:40.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.462+0000 7f6340ff9640 10 monclient: _finish_auth 0 2024-03-25T18:10:40.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.462+0000 7f6340ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:10:40.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.462+0000 7f6340ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:40.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.462+0000 7f634a4ed640 5 monclient: authenticate success, global_id 4525 2024-03-25T18:10:40.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.462+0000 7f634a4ed640 10 monclient: _renew_subs 2024-03-25T18:10:40.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.462+0000 7f634a4ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:10:40.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.462+0000 7f634a4ed640 1 -- 172.21.15.66:0/3309937963 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6344143bb0 con 0x7f63440b1730 2024-03-25T18:10:40.466 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.462+0000 7f634a4ed640 10 client.4525.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:40.466 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.462+0000 7f634a4ed640 10 monclient: _renew_subs 2024-03-25T18:10:40.466 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.462+0000 7f634a4ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:10:40.466 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.462+0000 7f634a4ed640 1 -- 172.21.15.66:0/3309937963 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6344130060 con 0x7f63440b1730 2024-03-25T18:10:40.466 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.463+0000 7f634a4ed640 1 librados: init done 2024-03-25T18:10:40.466 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.463+0000 7f634a4ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:40.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.463+0000 7f634a4ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:40.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.463+0000 7f634a4ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:10:40.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.463+0000 7f634a4ed640 1 -- 172.21.15.66:0/3309937963 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63440627b0 con 0x7f63440b1730 2024-03-25T18:10:40.499 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.498+0000 7f6340ff9640 1 -- 172.21.15.66:0/3309937963 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6334004940 con 0x7f63440b1730 2024-03-25T18:10:40.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.498+0000 7f6340ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:40.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.498+0000 7f6340ff9640 1 -- 172.21.15.66:0/3309937963 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f6334004c50 con 0x7f63440b1730 2024-03-25T18:10:40.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.498+0000 7f6340ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:40.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.498+0000 7f6340ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:40.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.498+0000 7f63427fc640 4 set_mon_vals no callback set 2024-03-25T18:10:40.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.498+0000 7f6340ff9640 10 monclient: dump: 2024-03-25T18:10:40.500 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:40.500 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:40.500 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:40.500 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:40.500 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:40.500 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:40.500 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:40.501 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:40.501 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:40.501 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:40.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.499+0000 7f6340ff9640 1 -- 172.21.15.66:0/3309937963 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f633401ac20 con 0x7f63440b1730 2024-03-25T18:10:40.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.499+0000 7f63437fe640 10 monclient: get_auth_request con 0x7f633006ddc0 auth_method 0 2024-03-25T18:10:40.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.500+0000 7f6340ff9640 10 client.4525.objecter ms_handle_connect 0x7f633006ddc0 2024-03-25T18:10:40.502 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.501+0000 7f6340ff9640 1 -- 172.21.15.66:0/3309937963 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f63340547f0 con 0x7f63440b1730 2024-03-25T18:10:40.502 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.501+0000 7f6340ff9640 10 client.4525.objecter ms_dispatch 0x7f6344000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:40.502 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.501+0000 7f6340ff9640 3 client.4525.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:40.502 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.501+0000 7f6340ff9640 3 client.4525.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:40.503 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.501+0000 7f6340ff9640 20 client.4525.objecter dump_active .. 0 homeless 2024-03-25T18:10:40.778 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.777+0000 7f6340ff9640 1 -- 172.21.15.66:0/3309937963 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6334050130 con 0x7f63440b1730 2024-03-25T18:10:40.778 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.777+0000 7f6340ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:40.779 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:40.777+0000 7f6340ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:41.050 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:41.046+0000 7f634a4ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:41.050 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:41.046+0000 7f634a4ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:41.050 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:41.046+0000 7f634a4ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:10:41.050 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:41.047+0000 7f634a4ed640 1 -- 172.21.15.66:0/3309937963 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6344069f40 con 0x7f63440b1730 2024-03-25T18:10:41.074 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:41.073+0000 7f6340ff9640 1 -- 172.21.15.66:0/3309937963 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f6334052230 con 0x7f63440b1730 2024-03-25T18:10:41.074 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:41.073+0000 7f6340ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:41.074 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:41.073+0000 7f6340ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:41.075 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:10:41.075 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:41.077 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:41.075+0000 7f634a4ed640 10 librados: watch_flush enter 2024-03-25T18:10:41.077 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:41.075+0000 7f634a4ed640 10 librados: watch_flush exit 2024-03-25T18:10:41.077 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:41.075+0000 7f634a4ed640 20 client.4525.objecter shutdown clearing up homeless session... 2024-03-25T18:10:41.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:41.075+0000 7f634a4ed640 10 client.4525.objecter successfully canceled tick 2024-03-25T18:10:41.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:41.075+0000 7f634a4ed640 1 -- 172.21.15.66:0/3309937963 >> v1:172.21.15.193:6800/2275676803 conn(0x7f633006ddc0 legacy=0x7f6330070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:41.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:41.075+0000 7f634a4ed640 10 monclient: shutdown 2024-03-25T18:10:41.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:41.075+0000 7f634a4ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:41.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:41.075+0000 7f634a4ed640 1 -- 172.21.15.66:0/3309937963 >> v1:172.21.15.193:6789/0 conn(0x7f63440b1730 legacy=0x7f6344136d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:41.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:41.075+0000 7f634a4ed640 1 -- 172.21.15.66:0/3309937963 shutdown_connections 2024-03-25T18:10:41.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:41.075+0000 7f634a4ed640 1 -- 172.21.15.66:0/3309937963 >> 172.21.15.66:0/3309937963 conn(0x7f63440617a0 msgr2=0x7f634413f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:41.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:41.076+0000 7f634a4ed640 1 -- 172.21.15.66:0/3309937963 shutdown_connections 2024-03-25T18:10:41.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:41.076+0000 7f634a4ed640 1 -- 172.21.15.66:0/3309937963 wait complete. 2024-03-25T18:10:41.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:41.076+0000 7f634a4ed640 1 librados: shutdown 2024-03-25T18:10:41.091 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:47.898 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:48.021 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.019+0000 7f8f4d54b640 10 monclient: get_monmap_and_config 2024-03-25T18:10:48.021 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.019+0000 7f8f4d54b640 10 monclient: build_initial_monmap 2024-03-25T18:10:48.021 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.019+0000 7f8f4d54b640 10 monclient: monmap: 2024-03-25T18:10:48.021 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:10:48.021 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:48.021 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:10:48.021391+0000 2024-03-25T18:10:48.022 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:10:48.021391+0000 2024-03-25T18:10:48.022 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:48.022 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:48.022 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:10:48.022 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:10:48.022 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:10:48.022 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:48.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.020+0000 7f8f4d54b640 1 Processor -- start 2024-03-25T18:10:48.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.020+0000 7f8f4d54b640 1 -- start start 2024-03-25T18:10:48.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.020+0000 7f8f4d54b640 10 monclient: init 2024-03-25T18:10:48.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.021+0000 7f8f4d54b640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:48.024 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.021+0000 7f8f4d54b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:10:48.024 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.021+0000 7f8f4d54b640 10 monclient(hunting): picked mon.noname-c con 0x7f8f480a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:10:48.024 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.021+0000 7f8f4d54b640 10 monclient(hunting): picked mon.noname-a con 0x7f8f4813f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:10:48.024 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.021+0000 7f8f4d54b640 10 monclient(hunting): picked mon.noname-b con 0x7f8f481430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:10:48.024 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.021+0000 7f8f4d54b640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f48142520 con 0x7f8f4813f710 2024-03-25T18:10:48.024 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.021+0000 7f8f4d54b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f480b28a0 con 0x7f8f480a98f0 2024-03-25T18:10:48.024 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.021+0000 7f8f4d54b640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f480b2a30 con 0x7f8f481430c0 2024-03-25T18:10:48.024 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.021+0000 7f8f4d54b640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:48.024 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.021+0000 7f8f4d54b640 10 monclient(hunting): authenticate will time out at 7397.081438s 2024-03-25T18:10:48.025 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.023+0000 7f8f477fe640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f8f481430c0 0x7f8f4814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:37164/0 (socket says 172.21.15.66:37164) 2024-03-25T18:10:48.025 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.023+0000 7f8f477fe640 1 -- 172.21.15.66:0/3737296992 learned_addr learned my addr 172.21.15.66:0/3737296992 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:10:48.033 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.031+0000 7f8f45ffb640 1 -- 172.21.15.66:0/3737296992 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1475647027 0 0) 0x7f8f480b2a30 con 0x7f8f481430c0 2024-03-25T18:10:48.033 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.031+0000 7f8f45ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:48.033 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.031+0000 7f8f45ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:48.033 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.031+0000 7f8f45ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:48.033 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.031+0000 7f8f45ffb640 10 monclient(hunting): my global_id is 4529 2024-03-25T18:10:48.033 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.031+0000 7f8f45ffb640 1 -- 172.21.15.66:0/3737296992 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f30003520 con 0x7f8f481430c0 2024-03-25T18:10:48.043 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.041+0000 7f8f45ffb640 1 -- 172.21.15.66:0/3737296992 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3046023674 0 0) 0x7f8f30003520 con 0x7f8f481430c0 2024-03-25T18:10:48.043 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.041+0000 7f8f45ffb640 1 -- 172.21.15.66:0/3737296992 >> v1:172.21.15.193:6789/0 conn(0x7f8f480a98f0 legacy=0x7f8f480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:10:48.043 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.041+0000 7f8f45ffb640 1 -- 172.21.15.66:0/3737296992 >> v1:172.21.15.66:6789/0 conn(0x7f8f4813f710 legacy=0x7f8f481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:10:48.043 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.041+0000 7f8f45ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:10:48.044 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.041+0000 7f8f45ffb640 1 monclient: found mon.noname-b 2024-03-25T18:10:48.044 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.041+0000 7f8f45ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:48.044 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.041+0000 7f8f45ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.66:6790/0 2024-03-25T18:10:48.044 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.041+0000 7f8f45ffb640 1 -- 172.21.15.66:0/3737296992 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f480b44c0 con 0x7f8f481430c0 2024-03-25T18:10:48.044 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.041+0000 7f8f45ffb640 10 monclient: _finish_auth 0 2024-03-25T18:10:48.044 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.041+0000 7f8f45ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:10:48.044 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.041+0000 7f8f45ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:48.044 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.041+0000 7f8f45ffb640 1 -- 172.21.15.66:0/3737296992 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8f40001690 con 0x7f8f481430c0 2024-03-25T18:10:48.044 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.041+0000 7f8f4d54b640 5 monclient: authenticate success, global_id 4529 2024-03-25T18:10:48.044 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.041+0000 7f8f4d54b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:48.044 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.041+0000 7f8f45ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:48.044 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.041+0000 7f8f45ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:48.044 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.041+0000 7f8f45ffb640 10 monclient: dump: 2024-03-25T18:10:48.044 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:48.045 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:48.045 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:48.045 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:48.045 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:48.045 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:48.045 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:48.045 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:48.045 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:48.045 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:48.045 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.041+0000 7f8f4d54b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:48.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.049+0000 7f8f45ffb640 1 -- 172.21.15.66:0/3737296992 <== mon.2 v1:172.21.15.66:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f40001860 con 0x7f8f481430c0 2024-03-25T18:10:48.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.049+0000 7f8f45ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:48.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.049+0000 7f8f45ffb640 1 -- 172.21.15.66:0/3737296992 <== mon.2 v1:172.21.15.66:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8f40001cb0 con 0x7f8f481430c0 2024-03-25T18:10:48.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.049+0000 7f8f4d54b640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:48.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.049+0000 7f8f4d54b640 4 set_mon_vals no callback set 2024-03-25T18:10:48.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.049+0000 7f8f45ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:48.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.049+0000 7f8f45ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:48.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.049+0000 7f8f45ffb640 10 monclient: dump: 2024-03-25T18:10:48.051 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:48.051 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:48.052 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:48.052 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:48.052 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:48.052 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:48.052 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:48.052 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:48.052 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:48.052 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:48.053 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.050+0000 7f8f4d54b640 10 monclient: shutdown 2024-03-25T18:10:48.053 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.050+0000 7f8f4d54b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:48.053 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.050+0000 7f8f4d54b640 1 -- 172.21.15.66:0/3737296992 >> v1:172.21.15.66:6790/0 conn(0x7f8f481430c0 legacy=0x7f8f4814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:48.054 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.050+0000 7f8f4d54b640 1 -- 172.21.15.66:0/3737296992 shutdown_connections 2024-03-25T18:10:48.054 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.050+0000 7f8f4d54b640 1 -- 172.21.15.66:0/3737296992 >> 172.21.15.66:0/3737296992 conn(0x7f8f480b7ea0 msgr2=0x7f8f480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:48.054 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.050+0000 7f8f4d54b640 1 -- 172.21.15.66:0/3737296992 shutdown_connections 2024-03-25T18:10:48.054 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.050+0000 7f8f4d54b640 1 -- 172.21.15.66:0/3737296992 wait complete. 2024-03-25T18:10:48.054 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.051+0000 7f8f4d54b640 10 monclient: build_initial_monmap 2024-03-25T18:10:48.054 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.051+0000 7f8f4d54b640 10 monclient: monmap: 2024-03-25T18:10:48.054 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:10:48.054 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:48.054 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:10:48.054 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:10:48.054 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:48.055 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:10:48.055 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:10:48.055 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:10:48.055 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:10:48.055 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:48.055 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.051+0000 7f8f4d54b640 1 librados: starting msgr at 2024-03-25T18:10:48.055 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.051+0000 7f8f4d54b640 1 librados: starting objecter 2024-03-25T18:10:48.055 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.052+0000 7f8f4d54b640 1 Processor -- start 2024-03-25T18:10:48.055 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.052+0000 7f8f4d54b640 1 -- start start 2024-03-25T18:10:48.055 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.052+0000 7f8f4d54b640 1 librados: setting wanted keys 2024-03-25T18:10:48.055 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.052+0000 7f8f4d54b640 1 librados: calling monclient init 2024-03-25T18:10:48.055 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.052+0000 7f8f4d54b640 10 monclient: init 2024-03-25T18:10:48.057 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.052+0000 7f8f4d54b640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:48.057 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.052+0000 7f8f4d54b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:10:48.057 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.052+0000 7f8f4d54b640 10 monclient(hunting): picked mon.noname-b con 0x7f8f480a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:10:48.057 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.052+0000 7f8f4d54b640 10 monclient(hunting): picked mon.noname-c con 0x7f8f4813f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:10:48.057 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.052+0000 7f8f4d54b640 10 monclient(hunting): picked mon.noname-a con 0x7f8f481430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:10:48.057 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.052+0000 7f8f4d54b640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f480abb90 con 0x7f8f481430c0 2024-03-25T18:10:48.057 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.052+0000 7f8f4d54b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f481da8f0 con 0x7f8f480a98f0 2024-03-25T18:10:48.057 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.053+0000 7f8f4d54b640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f481db7c0 con 0x7f8f4813f710 2024-03-25T18:10:48.057 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.053+0000 7f8f4d54b640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:48.057 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.053+0000 7f8f4d54b640 10 monclient(hunting): authenticate will time out at 7397.113118s 2024-03-25T18:10:48.063 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.061+0000 7f8f46ffd640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f8f480a98f0 0x7f8f481db0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:60974/0 (socket says 172.21.15.66:60974) 2024-03-25T18:10:48.063 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.062+0000 7f8f46ffd640 1 -- 172.21.15.66:0/1331539660 learned_addr learned my addr 172.21.15.66:0/1331539660 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:10:48.066 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.065+0000 7f8f2f7fe640 10 client.?.objecter ms_handle_connect 0x7f8f480a98f0 2024-03-25T18:10:48.067 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.065+0000 7f8f2f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:48.067 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.065+0000 7f8f2f7fe640 10 client.?.objecter ms_handle_connect 0x7f8f481430c0 2024-03-25T18:10:48.067 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.065+0000 7f8f2f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:48.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.077+0000 7f8f2f7fe640 10 client.?.objecter ms_handle_connect 0x7f8f4813f710 2024-03-25T18:10:48.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.077+0000 7f8f2f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:48.081 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.079+0000 7f8f2f7fe640 1 -- 172.21.15.66:0/1331539660 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3440441509 0 0) 0x7f8f481da8f0 con 0x7f8f480a98f0 2024-03-25T18:10:48.081 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.079+0000 7f8f2f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:10:48.081 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.079+0000 7f8f2f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:48.081 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.079+0000 7f8f2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:48.081 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.079+0000 7f8f2f7fe640 10 monclient(hunting): my global_id is 4528 2024-03-25T18:10:48.081 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.079+0000 7f8f2f7fe640 1 -- 172.21.15.66:0/1331539660 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f34004eb0 con 0x7f8f480a98f0 2024-03-25T18:10:48.082 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.080+0000 7f8f2f7fe640 1 -- 172.21.15.66:0/1331539660 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2314291124 0 0) 0x7f8f480abb90 con 0x7f8f481430c0 2024-03-25T18:10:48.082 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.080+0000 7f8f2f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:10:48.082 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.080+0000 7f8f2f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:48.082 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.080+0000 7f8f2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:48.082 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.080+0000 7f8f2f7fe640 10 monclient(hunting): my global_id is 4533 2024-03-25T18:10:48.082 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.080+0000 7f8f2f7fe640 1 -- 172.21.15.66:0/1331539660 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f481da8f0 con 0x7f8f481430c0 2024-03-25T18:10:48.104 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.102+0000 7f8f2f7fe640 1 -- 172.21.15.66:0/1331539660 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194306129 0 0) 0x7f8f481db7c0 con 0x7f8f4813f710 2024-03-25T18:10:48.104 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.102+0000 7f8f2f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:10:48.104 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.102+0000 7f8f2f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:48.104 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.102+0000 7f8f2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:48.104 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.102+0000 7f8f2f7fe640 10 monclient(hunting): my global_id is 4532 2024-03-25T18:10:48.104 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.102+0000 7f8f2f7fe640 1 -- 172.21.15.66:0/1331539660 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f480abb90 con 0x7f8f4813f710 2024-03-25T18:10:48.104 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.102+0000 7f8f2f7fe640 1 -- 172.21.15.66:0/1331539660 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3163534801 0 0) 0x7f8f481da8f0 con 0x7f8f481430c0 2024-03-25T18:10:48.104 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.102+0000 7f8f2f7fe640 1 -- 172.21.15.66:0/1331539660 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f481db7c0 con 0x7f8f481430c0 2024-03-25T18:10:48.104 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.102+0000 7f8f2f7fe640 1 -- 172.21.15.66:0/1331539660 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8f40001b70 con 0x7f8f481430c0 2024-03-25T18:10:48.104 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.102+0000 7f8f2f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:48.104 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.102+0000 7f8f2f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:10:48.104 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.102+0000 7f8f2f7fe640 10 monclient(hunting): dump: 2024-03-25T18:10:48.105 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:48.105 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:48.105 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:48.105 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:48.105 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:48.105 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:48.105 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:48.105 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:48.105 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:48.105 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:48.105 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.102+0000 7f8f2f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:48.113 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.112+0000 7f8f2f7fe640 1 -- 172.21.15.66:0/1331539660 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 732506992 0 0) 0x7f8f34004eb0 con 0x7f8f480a98f0 2024-03-25T18:10:48.113 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.112+0000 7f8f2f7fe640 1 -- 172.21.15.66:0/1331539660 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f481da8f0 con 0x7f8f480a98f0 2024-03-25T18:10:48.113 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.112+0000 7f8f2f7fe640 1 -- 172.21.15.66:0/1331539660 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8f38001ce0 con 0x7f8f480a98f0 2024-03-25T18:10:48.113 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.112+0000 7f8f2f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:48.114 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.112+0000 7f8f2f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:48.114 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.112+0000 7f8f2f7fe640 10 monclient(hunting): dump: 2024-03-25T18:10:48.114 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:48.114 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:48.114 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:48.114 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:48.114 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:48.114 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:48.114 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:48.114 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:48.114 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:48.114 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:48.120 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.119+0000 7f8f2f7fe640 1 -- 172.21.15.66:0/1331539660 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2579258500 0 0) 0x7f8f480abb90 con 0x7f8f4813f710 2024-03-25T18:10:48.120 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.119+0000 7f8f2f7fe640 1 -- 172.21.15.66:0/1331539660 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f34004eb0 con 0x7f8f4813f710 2024-03-25T18:10:48.121 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.119+0000 7f8f2f7fe640 1 -- 172.21.15.66:0/1331539660 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8f3c0017d0 con 0x7f8f4813f710 2024-03-25T18:10:48.121 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.119+0000 7f8f2f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:48.121 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.119+0000 7f8f2f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:48.121 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.119+0000 7f8f2f7fe640 10 monclient(hunting): dump: 2024-03-25T18:10:48.121 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:48.121 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:48.121 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:48.121 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:48.121 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:48.121 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:48.121 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:48.121 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:48.121 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:48.121 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:48.133 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.130+0000 7f8f2f7fe640 1 -- 172.21.15.66:0/1331539660 <== mon.0 v1:172.21.15.66:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1725220519 0 0) 0x7f8f481db7c0 con 0x7f8f481430c0 2024-03-25T18:10:48.133 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.130+0000 7f8f2f7fe640 1 -- 172.21.15.66:0/1331539660 >> v1:172.21.15.66:6790/0 conn(0x7f8f4813f710 legacy=0x7f8f48138630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:48.133 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.131+0000 7f8f2f7fe640 1 -- 172.21.15.66:0/1331539660 >> v1:172.21.15.193:6789/0 conn(0x7f8f480a98f0 legacy=0x7f8f481db0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:48.133 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.131+0000 7f8f2f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:10:48.133 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.131+0000 7f8f2f7fe640 1 monclient: found mon.a 2024-03-25T18:10:48.133 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.131+0000 7f8f2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:48.133 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.131+0000 7f8f2f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:10:48.133 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.131+0000 7f8f2f7fe640 1 -- 172.21.15.66:0/1331539660 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f481dab90 con 0x7f8f481430c0 2024-03-25T18:10:48.133 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.131+0000 7f8f2f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:10:48.133 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.131+0000 7f8f2f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:10:48.134 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.131+0000 7f8f2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:48.134 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.131+0000 7f8f4d54b640 5 monclient: authenticate success, global_id 4533 2024-03-25T18:10:48.134 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.131+0000 7f8f4d54b640 10 monclient: _renew_subs 2024-03-25T18:10:48.134 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.131+0000 7f8f4d54b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:10:48.134 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.131+0000 7f8f4d54b640 1 -- 172.21.15.66:0/1331539660 --> v1:172.21.15.66:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f48138d40 con 0x7f8f481430c0 2024-03-25T18:10:48.134 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.131+0000 7f8f4d54b640 10 client.4533.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:48.134 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.131+0000 7f8f4d54b640 10 monclient: _renew_subs 2024-03-25T18:10:48.134 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.131+0000 7f8f4d54b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:10:48.134 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.131+0000 7f8f4d54b640 1 -- 172.21.15.66:0/1331539660 --> v1:172.21.15.66:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f48137c00 con 0x7f8f481430c0 2024-03-25T18:10:48.134 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.131+0000 7f8f4d54b640 1 librados: init done 2024-03-25T18:10:48.135 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.132+0000 7f8f4d54b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:48.135 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.132+0000 7f8f4d54b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:48.135 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.132+0000 7f8f4d54b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:10:48.135 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.132+0000 7f8f4d54b640 1 -- 172.21.15.66:0/1331539660 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f4814f320 con 0x7f8f481430c0 2024-03-25T18:10:48.490 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.488+0000 7f8f2f7fe640 1 -- 172.21.15.66:0/1331539660 <== mon.0 v1:172.21.15.66:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f40001d10 con 0x7f8f481430c0 2024-03-25T18:10:48.490 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.488+0000 7f8f2f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:48.490 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.488+0000 7f8f2f7fe640 1 -- 172.21.15.66:0/1331539660 <== mon.0 v1:172.21.15.66:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8f40004c30 con 0x7f8f481430c0 2024-03-25T18:10:48.490 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.488+0000 7f8f2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:48.490 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.488+0000 7f8f2f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:48.490 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.488+0000 7f8f2f7fe640 10 monclient: dump: 2024-03-25T18:10:48.490 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:48.491 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:48.491 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:48.491 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:48.491 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:48.491 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:48.491 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:48.491 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:48.491 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:48.491 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:48.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.488+0000 7f8f457fa640 4 set_mon_vals no callback set 2024-03-25T18:10:48.492 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.489+0000 7f8f2f7fe640 1 -- 172.21.15.66:0/1331539660 <== mon.0 v1:172.21.15.66:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f8f4001ac00 con 0x7f8f481430c0 2024-03-25T18:10:48.492 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.490+0000 7f8f46ffd640 10 monclient: get_auth_request con 0x7f8f3406e0b0 auth_method 0 2024-03-25T18:10:48.492 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.490+0000 7f8f2f7fe640 10 client.4533.objecter ms_handle_connect 0x7f8f3406e0b0 2024-03-25T18:10:48.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.496+0000 7f8f2f7fe640 1 -- 172.21.15.66:0/1331539660 <== mon.0 v1:172.21.15.66:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f8f400546f0 con 0x7f8f481430c0 2024-03-25T18:10:48.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.496+0000 7f8f2f7fe640 10 client.4533.objecter ms_dispatch 0x7f8f48000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:48.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.496+0000 7f8f2f7fe640 3 client.4533.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:48.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.496+0000 7f8f2f7fe640 3 client.4533.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:48.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.496+0000 7f8f2f7fe640 20 client.4533.objecter dump_active .. 0 homeless 2024-03-25T18:10:48.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.496+0000 7f8f2f7fe640 1 -- 172.21.15.66:0/1331539660 <== mon.0 v1:172.21.15.66:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8f40050200 con 0x7f8f481430c0 2024-03-25T18:10:48.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.496+0000 7f8f2f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:48.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.496+0000 7f8f2f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:48.723 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.721+0000 7f8f4d54b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:48.723 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.721+0000 7f8f4d54b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:48.723 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.721+0000 7f8f4d54b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:10:48.723 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.721+0000 7f8f4d54b640 1 -- 172.21.15.66:0/1331539660 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f481dad20 con 0x7f8f481430c0 2024-03-25T18:10:48.759 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.757+0000 7f8f2f7fe640 1 -- 172.21.15.66:0/1331539660 <== mon.0 v1:172.21.15.66:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f8f40052130 con 0x7f8f481430c0 2024-03-25T18:10:48.759 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.757+0000 7f8f2f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:48.759 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.757+0000 7f8f2f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:48.759 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:10:48.759 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:48.761 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.759+0000 7f8f4d54b640 10 librados: watch_flush enter 2024-03-25T18:10:48.761 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.759+0000 7f8f4d54b640 10 librados: watch_flush exit 2024-03-25T18:10:48.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.759+0000 7f8f4d54b640 20 client.4533.objecter shutdown clearing up homeless session... 2024-03-25T18:10:48.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.759+0000 7f8f4d54b640 10 client.4533.objecter successfully canceled tick 2024-03-25T18:10:48.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.759+0000 7f8f4d54b640 1 -- 172.21.15.66:0/1331539660 >> v1:172.21.15.193:6800/2275676803 conn(0x7f8f3406e0b0 legacy=0x7f8f34070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:48.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.759+0000 7f8f4d54b640 10 monclient: shutdown 2024-03-25T18:10:48.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.759+0000 7f8f4d54b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:48.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.759+0000 7f8f4d54b640 1 -- 172.21.15.66:0/1331539660 >> v1:172.21.15.66:6789/0 conn(0x7f8f481430c0 legacy=0x7f8f48133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:48.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.759+0000 7f8f4d54b640 1 -- 172.21.15.66:0/1331539660 shutdown_connections 2024-03-25T18:10:48.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.759+0000 7f8f4d54b640 1 -- 172.21.15.66:0/1331539660 >> 172.21.15.66:0/1331539660 conn(0x7f8f480b7ea0 msgr2=0x7f8f48074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:48.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.760+0000 7f8f4d54b640 1 -- 172.21.15.66:0/1331539660 shutdown_connections 2024-03-25T18:10:48.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.760+0000 7f8f4d54b640 1 -- 172.21.15.66:0/1331539660 wait complete. 2024-03-25T18:10:48.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:48.760+0000 7f8f4d54b640 1 librados: shutdown 2024-03-25T18:10:48.774 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:49.801 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:07:20.781 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:49.807 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:07:20.787 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:49.849 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:07:20.829 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:49.860 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:07:20.818 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:49.867 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:07:20.848 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:49.868 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:07:20.827 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:49.881 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:07:20.861 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:49.893 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:07:20.851 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:49.909 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:07:20.889 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:55.580 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:55.701 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.700+0000 7f86e1aa5640 10 monclient: get_monmap_and_config 2024-03-25T18:10:55.702 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.700+0000 7f86e1aa5640 10 monclient: build_initial_monmap 2024-03-25T18:10:55.702 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.700+0000 7f86e1aa5640 10 monclient: monmap: 2024-03-25T18:10:55.702 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:10:55.702 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:55.702 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:10:55.701596+0000 2024-03-25T18:10:55.702 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:10:55.701596+0000 2024-03-25T18:10:55.702 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:55.702 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:55.702 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:10:55.702 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:10:55.702 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:10:55.702 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:55.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.701+0000 7f86e1aa5640 1 Processor -- start 2024-03-25T18:10:55.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.701+0000 7f86e1aa5640 1 -- start start 2024-03-25T18:10:55.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.701+0000 7f86e1aa5640 10 monclient: init 2024-03-25T18:10:55.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.701+0000 7f86e1aa5640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:55.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.701+0000 7f86e1aa5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:10:55.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.701+0000 7f86e1aa5640 10 monclient(hunting): picked mon.noname-c con 0x7f86dc0a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:10:55.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.701+0000 7f86e1aa5640 10 monclient(hunting): picked mon.noname-a con 0x7f86dc13f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:10:55.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.702+0000 7f86e1aa5640 10 monclient(hunting): picked mon.noname-b con 0x7f86dc1430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:10:55.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.702+0000 7f86e1aa5640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86dc142520 con 0x7f86dc13f710 2024-03-25T18:10:55.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.702+0000 7f86e1aa5640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86dc0b28a0 con 0x7f86dc0a98f0 2024-03-25T18:10:55.704 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.702+0000 7f86e1aa5640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86dc0b2a30 con 0x7f86dc1430c0 2024-03-25T18:10:55.705 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.702+0000 7f86e1aa5640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:55.705 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.702+0000 7f86e1aa5640 10 monclient(hunting): authenticate will time out at 7404.762304s 2024-03-25T18:10:55.710 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.709+0000 7f86daffd640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f86dc13f710 0x7f86dc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:37812/0 (socket says 172.21.15.66:37812) 2024-03-25T18:10:55.710 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.709+0000 7f86daffd640 1 -- 172.21.15.66:0/3569112548 learned_addr learned my addr 172.21.15.66:0/3569112548 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:10:55.723 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.721+0000 7f86da7fc640 1 -- 172.21.15.66:0/3569112548 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1054055100 0 0) 0x7f86dc142520 con 0x7f86dc13f710 2024-03-25T18:10:55.723 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.721+0000 7f86da7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:10:55.723 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.721+0000 7f86da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:55.723 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.721+0000 7f86da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:55.723 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.721+0000 7f86da7fc640 10 monclient(hunting): my global_id is 4536 2024-03-25T18:10:55.723 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.721+0000 7f86da7fc640 1 -- 172.21.15.66:0/3569112548 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c4003520 con 0x7f86dc13f710 2024-03-25T18:10:55.730 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.728+0000 7f86da7fc640 1 -- 172.21.15.66:0/3569112548 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1363829001 0 0) 0x7f86c4003520 con 0x7f86dc13f710 2024-03-25T18:10:55.730 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.728+0000 7f86da7fc640 1 -- 172.21.15.66:0/3569112548 >> v1:172.21.15.66:6790/0 conn(0x7f86dc1430c0 legacy=0x7f86dc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:10:55.730 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.728+0000 7f86da7fc640 1 -- 172.21.15.66:0/3569112548 >> v1:172.21.15.193:6789/0 conn(0x7f86dc0a98f0 legacy=0x7f86dc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:55.730 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.728+0000 7f86da7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:10:55.730 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.728+0000 7f86da7fc640 1 monclient: found mon.noname-a 2024-03-25T18:10:55.730 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.728+0000 7f86da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:55.730 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.728+0000 7f86da7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.66:6789/0 2024-03-25T18:10:55.730 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.728+0000 7f86da7fc640 1 -- 172.21.15.66:0/3569112548 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86dc0b44c0 con 0x7f86dc13f710 2024-03-25T18:10:55.730 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.728+0000 7f86da7fc640 10 monclient: _finish_auth 0 2024-03-25T18:10:55.730 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.728+0000 7f86da7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:10:55.731 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.728+0000 7f86da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:55.731 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.728+0000 7f86da7fc640 1 -- 172.21.15.66:0/3569112548 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f86d0001690 con 0x7f86dc13f710 2024-03-25T18:10:55.731 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.728+0000 7f86da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:55.731 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.728+0000 7f86da7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:10:55.731 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.728+0000 7f86da7fc640 10 monclient: dump: 2024-03-25T18:10:55.731 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:55.731 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:55.731 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:55.731 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:55.731 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:55.731 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:55.731 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:55.731 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:55.731 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:55.731 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:55.731 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.728+0000 7f86e1aa5640 5 monclient: authenticate success, global_id 4536 2024-03-25T18:10:55.732 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.728+0000 7f86e1aa5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:55.749 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.747+0000 7f86da7fc640 1 -- 172.21.15.66:0/3569112548 <== mon.0 v1:172.21.15.66:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86d0001860 con 0x7f86dc13f710 2024-03-25T18:10:55.749 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.748+0000 7f86da7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:55.749 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.748+0000 7f86da7fc640 1 -- 172.21.15.66:0/3569112548 <== mon.0 v1:172.21.15.66:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f86d0001c70 con 0x7f86dc13f710 2024-03-25T18:10:55.749 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.748+0000 7f86da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:55.750 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.748+0000 7f86da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:55.750 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.748+0000 7f86da7fc640 10 monclient: dump: 2024-03-25T18:10:55.750 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:55.750 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:55.750 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:55.750 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:55.750 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:55.750 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:55.750 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:55.750 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:55.750 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:55.750 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:55.750 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.748+0000 7f86e1aa5640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:55.750 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.748+0000 7f86e1aa5640 4 set_mon_vals no callback set 2024-03-25T18:10:55.753 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.749+0000 7f86e1aa5640 10 monclient: shutdown 2024-03-25T18:10:55.753 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.749+0000 7f86e1aa5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:55.753 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.749+0000 7f86e1aa5640 1 -- 172.21.15.66:0/3569112548 >> v1:172.21.15.66:6789/0 conn(0x7f86dc13f710 legacy=0x7f86dc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:55.753 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.749+0000 7f86e1aa5640 1 -- 172.21.15.66:0/3569112548 shutdown_connections 2024-03-25T18:10:55.753 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.749+0000 7f86e1aa5640 1 -- 172.21.15.66:0/3569112548 >> 172.21.15.66:0/3569112548 conn(0x7f86dc0b7ea0 msgr2=0x7f86dc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:55.753 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.749+0000 7f86e1aa5640 1 -- 172.21.15.66:0/3569112548 shutdown_connections 2024-03-25T18:10:55.753 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.749+0000 7f86e1aa5640 1 -- 172.21.15.66:0/3569112548 wait complete. 2024-03-25T18:10:55.753 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.750+0000 7f86e1aa5640 10 monclient: build_initial_monmap 2024-03-25T18:10:55.754 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.750+0000 7f86e1aa5640 10 monclient: monmap: 2024-03-25T18:10:55.754 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:10:55.754 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:55.754 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:10:55.754 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:10:55.754 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:55.754 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:10:55.754 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:10:55.754 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:10:55.754 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:10:55.754 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:55.754 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.750+0000 7f86e1aa5640 1 librados: starting msgr at 2024-03-25T18:10:55.754 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.750+0000 7f86e1aa5640 1 librados: starting objecter 2024-03-25T18:10:55.755 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.750+0000 7f86e1aa5640 1 Processor -- start 2024-03-25T18:10:55.755 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.750+0000 7f86e1aa5640 1 -- start start 2024-03-25T18:10:55.755 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.750+0000 7f86e1aa5640 1 librados: setting wanted keys 2024-03-25T18:10:55.755 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.750+0000 7f86e1aa5640 1 librados: calling monclient init 2024-03-25T18:10:55.755 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.750+0000 7f86e1aa5640 10 monclient: init 2024-03-25T18:10:55.755 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.751+0000 7f86e1aa5640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:55.755 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.751+0000 7f86e1aa5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:10:55.755 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.751+0000 7f86e1aa5640 10 monclient(hunting): picked mon.noname-b con 0x7f86dc0a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:10:55.755 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.751+0000 7f86e1aa5640 10 monclient(hunting): picked mon.noname-a con 0x7f86dc13f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:10:55.755 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.751+0000 7f86e1aa5640 10 monclient(hunting): picked mon.noname-c con 0x7f86dc1430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:10:55.755 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.751+0000 7f86e1aa5640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86dc0b0870 con 0x7f86dc13f710 2024-03-25T18:10:55.756 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.751+0000 7f86e1aa5640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86dc0b16c0 con 0x7f86dc0a98f0 2024-03-25T18:10:55.756 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.751+0000 7f86e1aa5640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86dc0b0ae0 con 0x7f86dc1430c0 2024-03-25T18:10:55.756 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.751+0000 7f86e1aa5640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:55.756 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.751+0000 7f86e1aa5640 10 monclient(hunting): authenticate will time out at 7404.811614s 2024-03-25T18:10:55.759 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.758+0000 7f86dbfff640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f86dc1430c0 0x7f86dc1e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:48794/0 (socket says 172.21.15.66:48794) 2024-03-25T18:10:55.759 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.758+0000 7f86dbfff640 1 -- 172.21.15.66:0/3205417291 learned_addr learned my addr 172.21.15.66:0/3205417291 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:10:55.760 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.759+0000 7f86c3fff640 10 client.?.objecter ms_handle_connect 0x7f86dc0a98f0 2024-03-25T18:10:55.760 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.759+0000 7f86c3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:55.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.761+0000 7f86c3fff640 10 client.?.objecter ms_handle_connect 0x7f86dc1430c0 2024-03-25T18:10:55.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.761+0000 7f86c3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:55.765 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.764+0000 7f86c3fff640 10 client.?.objecter ms_handle_connect 0x7f86dc13f710 2024-03-25T18:10:55.766 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.764+0000 7f86c3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:55.769 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.767+0000 7f86c3fff640 1 -- 172.21.15.66:0/3205417291 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1520278101 0 0) 0x7f86dc0b16c0 con 0x7f86dc0a98f0 2024-03-25T18:10:55.769 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.767+0000 7f86c3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:55.769 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.767+0000 7f86c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:55.769 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.767+0000 7f86c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:55.769 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.767+0000 7f86c3fff640 10 monclient(hunting): my global_id is 4534 2024-03-25T18:10:55.769 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.767+0000 7f86c3fff640 1 -- 172.21.15.66:0/3205417291 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c8004eb0 con 0x7f86dc0a98f0 2024-03-25T18:10:55.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.770+0000 7f86c3fff640 1 -- 172.21.15.66:0/3205417291 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927720846 0 0) 0x7f86dc0b0870 con 0x7f86dc13f710 2024-03-25T18:10:55.772 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.770+0000 7f86c3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:55.772 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.770+0000 7f86c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:55.772 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.770+0000 7f86c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:55.772 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.770+0000 7f86c3fff640 10 monclient(hunting): my global_id is 4539 2024-03-25T18:10:55.772 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.770+0000 7f86c3fff640 1 -- 172.21.15.66:0/3205417291 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86dc0b16c0 con 0x7f86dc13f710 2024-03-25T18:10:55.776 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.775+0000 7f86c3fff640 1 -- 172.21.15.66:0/3205417291 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2731027231 0 0) 0x7f86c8004eb0 con 0x7f86dc0a98f0 2024-03-25T18:10:55.777 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.775+0000 7f86c3fff640 1 -- 172.21.15.66:0/3205417291 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86dc0b0870 con 0x7f86dc0a98f0 2024-03-25T18:10:55.777 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.775+0000 7f86c3fff640 1 -- 172.21.15.66:0/3205417291 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f86cc001930 con 0x7f86dc0a98f0 2024-03-25T18:10:55.777 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.775+0000 7f86c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:55.777 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.775+0000 7f86c3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:55.777 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.775+0000 7f86c3fff640 10 monclient(hunting): dump: 2024-03-25T18:10:55.777 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:55.777 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:55.777 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:55.777 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:55.777 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:55.777 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:55.777 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:55.777 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:55.777 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:55.777 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:55.778 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.775+0000 7f86c3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:55.782 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.780+0000 7f86c3fff640 1 -- 172.21.15.66:0/3205417291 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2594000042 0 0) 0x7f86dc0b16c0 con 0x7f86dc13f710 2024-03-25T18:10:55.782 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.780+0000 7f86c3fff640 1 -- 172.21.15.66:0/3205417291 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86c8004eb0 con 0x7f86dc13f710 2024-03-25T18:10:55.782 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.780+0000 7f86c3fff640 1 -- 172.21.15.66:0/3205417291 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f86d0001b70 con 0x7f86dc13f710 2024-03-25T18:10:55.782 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.780+0000 7f86c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:55.782 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.780+0000 7f86c3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:55.782 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.780+0000 7f86c3fff640 10 monclient(hunting): dump: 2024-03-25T18:10:55.782 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:55.782 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:55.782 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:55.783 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:55.783 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:55.783 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:55.783 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:55.783 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:55.783 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:55.783 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:55.783 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.781+0000 7f86c3fff640 1 -- 172.21.15.66:0/3205417291 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 202224416 0 0) 0x7f86dc0b0ae0 con 0x7f86dc1430c0 2024-03-25T18:10:55.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.781+0000 7f86c3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:55.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.781+0000 7f86c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:55.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.781+0000 7f86c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:55.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.781+0000 7f86c3fff640 10 monclient(hunting): my global_id is 4535 2024-03-25T18:10:55.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.781+0000 7f86c3fff640 1 -- 172.21.15.66:0/3205417291 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86dc0b16c0 con 0x7f86dc1430c0 2024-03-25T18:10:55.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.788+0000 7f86c3fff640 1 -- 172.21.15.66:0/3205417291 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1384934403 0 0) 0x7f86dc0b0870 con 0x7f86dc0a98f0 2024-03-25T18:10:55.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.788+0000 7f86c3fff640 1 -- 172.21.15.66:0/3205417291 >> v1:172.21.15.66:6790/0 conn(0x7f86dc1430c0 legacy=0x7f86dc1e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:55.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.788+0000 7f86c3fff640 1 -- 172.21.15.66:0/3205417291 >> v1:172.21.15.66:6789/0 conn(0x7f86dc13f710 legacy=0x7f86dc1df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:55.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.788+0000 7f86c3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:10:55.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.788+0000 7f86c3fff640 1 monclient: found mon.b 2024-03-25T18:10:55.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.788+0000 7f86c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:55.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.788+0000 7f86c3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:10:55.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.788+0000 7f86c3fff640 1 -- 172.21.15.66:0/3205417291 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86dc0b0dc0 con 0x7f86dc0a98f0 2024-03-25T18:10:55.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.788+0000 7f86c3fff640 10 monclient: _finish_auth 0 2024-03-25T18:10:55.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.788+0000 7f86c3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:10:55.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.788+0000 7f86c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:55.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.788+0000 7f86e1aa5640 5 monclient: authenticate success, global_id 4534 2024-03-25T18:10:55.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.788+0000 7f86e1aa5640 10 monclient: _renew_subs 2024-03-25T18:10:55.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.788+0000 7f86e1aa5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:10:55.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.788+0000 7f86e1aa5640 1 -- 172.21.15.66:0/3205417291 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86dc0ae760 con 0x7f86dc0a98f0 2024-03-25T18:10:55.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.788+0000 7f86e1aa5640 10 client.4534.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:55.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.788+0000 7f86e1aa5640 10 monclient: _renew_subs 2024-03-25T18:10:55.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.788+0000 7f86e1aa5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:10:55.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.788+0000 7f86e1aa5640 1 -- 172.21.15.66:0/3205417291 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86dc0b1430 con 0x7f86dc0a98f0 2024-03-25T18:10:55.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.788+0000 7f86e1aa5640 1 librados: init done 2024-03-25T18:10:55.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.789+0000 7f86e1aa5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:55.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.789+0000 7f86e1aa5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:55.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.789+0000 7f86e1aa5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:10:55.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:55.789+0000 7f86e1aa5640 1 -- 172.21.15.66:0/3205417291 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86dc0627b0 con 0x7f86dc0a98f0 2024-03-25T18:10:56.076 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.074+0000 7f86c3fff640 1 -- 172.21.15.66:0/3205417291 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86cc0035f0 con 0x7f86dc0a98f0 2024-03-25T18:10:56.076 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.074+0000 7f86c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:56.076 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.074+0000 7f86c3fff640 1 -- 172.21.15.66:0/3205417291 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f86cc003980 con 0x7f86dc0a98f0 2024-03-25T18:10:56.076 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.074+0000 7f86c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:56.076 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.074+0000 7f86c3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:56.076 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.074+0000 7f86c3fff640 10 monclient: dump: 2024-03-25T18:10:56.076 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:10:56.076 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:10:56.076 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:56.076 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:10:56.077 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:56.077 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:10:56.077 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:10:56.077 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:10:56.077 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:10:56.077 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:10:56.077 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.074+0000 7f86d9ffb640 4 set_mon_vals no callback set 2024-03-25T18:10:56.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.075+0000 7f86c3fff640 1 -- 172.21.15.66:0/3205417291 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f86cc003ba0 con 0x7f86dc0a98f0 2024-03-25T18:10:56.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.076+0000 7f86daffd640 10 monclient: get_auth_request con 0x7f86c806df40 auth_method 0 2024-03-25T18:10:56.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.076+0000 7f86c3fff640 10 client.4534.objecter ms_handle_connect 0x7f86c806df40 2024-03-25T18:10:56.082 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.080+0000 7f86c3fff640 1 -- 172.21.15.66:0/3205417291 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f86cc0535c0 con 0x7f86dc0a98f0 2024-03-25T18:10:56.082 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.080+0000 7f86c3fff640 10 client.4534.objecter ms_dispatch 0x7f86dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:56.082 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.080+0000 7f86c3fff640 3 client.4534.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:56.082 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.080+0000 7f86c3fff640 3 client.4534.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:56.082 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.080+0000 7f86c3fff640 20 client.4534.objecter dump_active .. 0 homeless 2024-03-25T18:10:56.083 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.081+0000 7f86c3fff640 1 -- 172.21.15.66:0/3205417291 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f86cc04f0d0 con 0x7f86dc0a98f0 2024-03-25T18:10:56.083 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.081+0000 7f86c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:56.083 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.081+0000 7f86c3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:56.345 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.344+0000 7f86e1aa5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:56.346 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.344+0000 7f86e1aa5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:56.346 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.344+0000 7f86e1aa5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:10:56.346 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.344+0000 7f86e1aa5640 1 -- 172.21.15.66:0/3205417291 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86dc141de0 con 0x7f86dc0a98f0 2024-03-25T18:10:56.371 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.370+0000 7f86c3fff640 1 -- 172.21.15.66:0/3205417291 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f86cc051000 con 0x7f86dc0a98f0 2024-03-25T18:10:56.371 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.370+0000 7f86c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:56.371 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.370+0000 7f86c3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:56.371 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:10:56.372 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:56.374 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.371+0000 7f86e1aa5640 10 librados: watch_flush enter 2024-03-25T18:10:56.374 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.372+0000 7f86e1aa5640 10 librados: watch_flush exit 2024-03-25T18:10:56.374 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.372+0000 7f86e1aa5640 20 client.4534.objecter shutdown clearing up homeless session... 2024-03-25T18:10:56.374 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.372+0000 7f86e1aa5640 10 client.4534.objecter successfully canceled tick 2024-03-25T18:10:56.374 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.372+0000 7f86e1aa5640 1 -- 172.21.15.66:0/3205417291 >> v1:172.21.15.193:6800/2275676803 conn(0x7f86c806df40 legacy=0x7f86c8070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:56.375 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.372+0000 7f86e1aa5640 10 monclient: shutdown 2024-03-25T18:10:56.375 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.372+0000 7f86e1aa5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:56.375 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.372+0000 7f86e1aa5640 1 -- 172.21.15.66:0/3205417291 >> v1:172.21.15.193:6789/0 conn(0x7f86dc0a98f0 legacy=0x7f86dc0abd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:56.375 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.372+0000 7f86e1aa5640 1 -- 172.21.15.66:0/3205417291 shutdown_connections 2024-03-25T18:10:56.375 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.372+0000 7f86e1aa5640 1 -- 172.21.15.66:0/3205417291 >> 172.21.15.66:0/3205417291 conn(0x7f86dc0b7ea0 msgr2=0x7f86dc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:56.375 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.372+0000 7f86e1aa5640 1 -- 172.21.15.66:0/3205417291 shutdown_connections 2024-03-25T18:10:56.375 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.372+0000 7f86e1aa5640 1 -- 172.21.15.66:0/3205417291 wait complete. 2024-03-25T18:10:56.375 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:10:56.372+0000 7f86e1aa5640 1 librados: shutdown 2024-03-25T18:10:56.386 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:03.193 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:03.313 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.312+0000 7f032591c640 10 monclient: get_monmap_and_config 2024-03-25T18:11:03.313 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.312+0000 7f032591c640 10 monclient: build_initial_monmap 2024-03-25T18:11:03.313 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.312+0000 7f032591c640 10 monclient: monmap: 2024-03-25T18:11:03.314 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:11:03.314 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:03.314 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:11:03.313583+0000 2024-03-25T18:11:03.314 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:11:03.313583+0000 2024-03-25T18:11:03.314 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:03.314 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:03.314 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:11:03.314 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:11:03.314 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:11:03.314 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:03.315 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.312+0000 7f032591c640 1 Processor -- start 2024-03-25T18:11:03.316 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.312+0000 7f032591c640 1 -- start start 2024-03-25T18:11:03.316 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.312+0000 7f032591c640 10 monclient: init 2024-03-25T18:11:03.316 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.313+0000 7f032591c640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:03.316 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.313+0000 7f032591c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:11:03.316 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.313+0000 7f032591c640 10 monclient(hunting): picked mon.noname-c con 0x7f03200a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:11:03.316 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.313+0000 7f032591c640 10 monclient(hunting): picked mon.noname-a con 0x7f032013f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:11:03.316 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.313+0000 7f032591c640 10 monclient(hunting): picked mon.noname-b con 0x7f03201430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:11:03.316 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.313+0000 7f032591c640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0320142520 con 0x7f032013f710 2024-03-25T18:11:03.316 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.313+0000 7f032591c640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03200b28a0 con 0x7f03200a98f0 2024-03-25T18:11:03.316 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.313+0000 7f032591c640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03200b2a30 con 0x7f03201430c0 2024-03-25T18:11:03.316 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.313+0000 7f032591c640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:03.316 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.313+0000 7f032591c640 10 monclient(hunting): authenticate will time out at 7412.373789s 2024-03-25T18:11:03.319 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.318+0000 7f031f7fe640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f03201430c0 0x7f032014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:53408/0 (socket says 172.21.15.66:53408) 2024-03-25T18:11:03.319 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.318+0000 7f031f7fe640 1 -- 172.21.15.66:0/1473023210 learned_addr learned my addr 172.21.15.66:0/1473023210 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:11:03.332 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.331+0000 7f031e7fc640 1 -- 172.21.15.66:0/1473023210 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206399343 0 0) 0x7f03200b2a30 con 0x7f03201430c0 2024-03-25T18:11:03.332 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.331+0000 7f031e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:03.332 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.331+0000 7f031e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:03.332 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.331+0000 7f031e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:03.332 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.331+0000 7f031e7fc640 10 monclient(hunting): my global_id is 4538 2024-03-25T18:11:03.332 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.331+0000 7f031e7fc640 1 -- 172.21.15.66:0/1473023210 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f030c003520 con 0x7f03201430c0 2024-03-25T18:11:03.334 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.332+0000 7f031e7fc640 1 -- 172.21.15.66:0/1473023210 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308684432 0 0) 0x7f03200b28a0 con 0x7f03200a98f0 2024-03-25T18:11:03.334 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.332+0000 7f031e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:03.334 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.332+0000 7f031e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:03.334 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.332+0000 7f031e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:03.334 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.332+0000 7f031e7fc640 10 monclient(hunting): my global_id is 4537 2024-03-25T18:11:03.334 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.332+0000 7f031e7fc640 1 -- 172.21.15.66:0/1473023210 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03200b2a30 con 0x7f03200a98f0 2024-03-25T18:11:03.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.350+0000 7f031e7fc640 1 -- 172.21.15.66:0/1473023210 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2284065392 0 0) 0x7f03200b2a30 con 0x7f03200a98f0 2024-03-25T18:11:03.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.350+0000 7f031e7fc640 1 -- 172.21.15.66:0/1473023210 >> v1:172.21.15.66:6790/0 conn(0x7f03201430c0 legacy=0x7f032014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:03.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.350+0000 7f031e7fc640 1 -- 172.21.15.66:0/1473023210 >> v1:172.21.15.66:6789/0 conn(0x7f032013f710 legacy=0x7f03201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:03.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.350+0000 7f031e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:11:03.353 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.350+0000 7f031e7fc640 1 monclient: found mon.noname-c 2024-03-25T18:11:03.353 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.350+0000 7f031e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:03.353 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.350+0000 7f031e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:11:03.353 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.350+0000 7f031e7fc640 1 -- 172.21.15.66:0/1473023210 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03200b44c0 con 0x7f03200a98f0 2024-03-25T18:11:03.353 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.350+0000 7f031e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:11:03.353 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.350+0000 7f031e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:11:03.353 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.350+0000 7f031e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:03.353 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.350+0000 7f031e7fc640 1 -- 172.21.15.66:0/1473023210 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f0310001670 con 0x7f03200a98f0 2024-03-25T18:11:03.353 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.350+0000 7f031e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:03.353 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.350+0000 7f031e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:03.353 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.350+0000 7f031e7fc640 10 monclient: dump: 2024-03-25T18:11:03.353 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:03.353 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:03.353 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:03.353 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:03.354 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:03.354 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:03.354 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:03.354 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:03.354 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:03.354 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:03.354 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.350+0000 7f032591c640 5 monclient: authenticate success, global_id 4537 2024-03-25T18:11:03.354 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.351+0000 7f032591c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:03.357 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.356+0000 7f031e7fc640 1 -- 172.21.15.66:0/1473023210 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03100017d0 con 0x7f03200a98f0 2024-03-25T18:11:03.358 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.356+0000 7f031e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:03.358 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.356+0000 7f031e7fc640 1 -- 172.21.15.66:0/1473023210 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f0310001ba0 con 0x7f03200a98f0 2024-03-25T18:11:03.358 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.356+0000 7f032591c640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:03.358 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.356+0000 7f032591c640 4 set_mon_vals no callback set 2024-03-25T18:11:03.358 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.356+0000 7f031e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:03.358 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.356+0000 7f031e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:03.358 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.356+0000 7f031e7fc640 10 monclient: dump: 2024-03-25T18:11:03.358 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:03.358 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:03.358 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:03.358 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:03.358 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:03.358 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:03.359 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:03.359 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:03.359 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:03.359 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:03.360 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.357+0000 7f032591c640 10 monclient: shutdown 2024-03-25T18:11:03.360 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.357+0000 7f032591c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:03.360 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.357+0000 7f032591c640 1 -- 172.21.15.66:0/1473023210 >> v1:172.21.15.193:6789/0 conn(0x7f03200a98f0 legacy=0x7f03200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:03.360 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.357+0000 7f032591c640 1 -- 172.21.15.66:0/1473023210 shutdown_connections 2024-03-25T18:11:03.360 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.357+0000 7f032591c640 1 -- 172.21.15.66:0/1473023210 >> 172.21.15.66:0/1473023210 conn(0x7f03200b7ea0 msgr2=0x7f03200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:03.360 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.357+0000 7f032591c640 1 -- 172.21.15.66:0/1473023210 shutdown_connections 2024-03-25T18:11:03.360 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.357+0000 7f032591c640 1 -- 172.21.15.66:0/1473023210 wait complete. 2024-03-25T18:11:03.360 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.358+0000 7f032591c640 10 monclient: build_initial_monmap 2024-03-25T18:11:03.361 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.358+0000 7f032591c640 10 monclient: monmap: 2024-03-25T18:11:03.361 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:11:03.361 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:03.361 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:11:03.361 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:11:03.361 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:03.361 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:11:03.361 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:11:03.361 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:11:03.361 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:11:03.361 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:03.361 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.359+0000 7f032591c640 1 librados: starting msgr at 2024-03-25T18:11:03.361 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.359+0000 7f032591c640 1 librados: starting objecter 2024-03-25T18:11:03.361 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.359+0000 7f032591c640 1 Processor -- start 2024-03-25T18:11:03.363 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.359+0000 7f032591c640 1 -- start start 2024-03-25T18:11:03.363 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.359+0000 7f032591c640 1 librados: setting wanted keys 2024-03-25T18:11:03.363 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.359+0000 7f032591c640 1 librados: calling monclient init 2024-03-25T18:11:03.363 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.359+0000 7f032591c640 10 monclient: init 2024-03-25T18:11:03.363 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.360+0000 7f032591c640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:03.363 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.360+0000 7f032591c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:11:03.363 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.360+0000 7f032591c640 10 monclient(hunting): picked mon.noname-c con 0x7f03200a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:11:03.363 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.360+0000 7f032591c640 10 monclient(hunting): picked mon.noname-a con 0x7f032013f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:11:03.363 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.360+0000 7f032591c640 10 monclient(hunting): picked mon.noname-b con 0x7f03201430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:11:03.363 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.360+0000 7f032591c640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f032012d970 con 0x7f032013f710 2024-03-25T18:11:03.363 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.360+0000 7f032591c640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03200b7250 con 0x7f03201430c0 2024-03-25T18:11:03.364 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.360+0000 7f032591c640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f032013b460 con 0x7f03200a98f0 2024-03-25T18:11:03.364 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.360+0000 7f032591c640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:03.364 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.360+0000 7f032591c640 10 monclient(hunting): authenticate will time out at 7412.420515s 2024-03-25T18:11:03.372 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.370+0000 7f031effd640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f03200a98f0 0x7f0320139d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:53456/0 (socket says 172.21.15.66:53456) 2024-03-25T18:11:03.372 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.370+0000 7f031effd640 1 -- 172.21.15.66:0/3991597129 learned_addr learned my addr 172.21.15.66:0/3991597129 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:11:03.381 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.380+0000 7f031cff9640 10 client.?.objecter ms_handle_connect 0x7f03201430c0 2024-03-25T18:11:03.381 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.380+0000 7f031cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:03.382 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.381+0000 7f031cff9640 10 client.?.objecter ms_handle_connect 0x7f03200a98f0 2024-03-25T18:11:03.382 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.381+0000 7f031cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:03.384 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.383+0000 7f031cff9640 10 client.?.objecter ms_handle_connect 0x7f032013f710 2024-03-25T18:11:03.384 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.383+0000 7f031cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:03.389 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.388+0000 7f031cff9640 1 -- 172.21.15.66:0/3991597129 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3270919081 0 0) 0x7f032013b460 con 0x7f03200a98f0 2024-03-25T18:11:03.389 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.388+0000 7f031cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:11:03.389 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.388+0000 7f031cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:03.389 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.388+0000 7f031cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:03.389 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.388+0000 7f031cff9640 10 monclient(hunting): my global_id is 4541 2024-03-25T18:11:03.390 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.388+0000 7f031cff9640 1 -- 172.21.15.66:0/3991597129 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0300004eb0 con 0x7f03200a98f0 2024-03-25T18:11:03.390 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.389+0000 7f031cff9640 1 -- 172.21.15.66:0/3991597129 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4119707076 0 0) 0x7f03200b7250 con 0x7f03201430c0 2024-03-25T18:11:03.390 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.389+0000 7f031cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:11:03.390 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.389+0000 7f031cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:03.390 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.389+0000 7f031cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:03.390 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.389+0000 7f031cff9640 10 monclient(hunting): my global_id is 4540 2024-03-25T18:11:03.390 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.389+0000 7f031cff9640 1 -- 172.21.15.66:0/3991597129 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f032013b460 con 0x7f03201430c0 2024-03-25T18:11:03.394 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.392+0000 7f031cff9640 1 -- 172.21.15.66:0/3991597129 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222449795 0 0) 0x7f032012d970 con 0x7f032013f710 2024-03-25T18:11:03.394 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.392+0000 7f031cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:11:03.394 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.392+0000 7f031cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:03.394 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.392+0000 7f031cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:03.394 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.392+0000 7f031cff9640 10 monclient(hunting): my global_id is 4545 2024-03-25T18:11:03.394 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.392+0000 7f031cff9640 1 -- 172.21.15.66:0/3991597129 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03200b7250 con 0x7f032013f710 2024-03-25T18:11:03.409 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.407+0000 7f031cff9640 1 -- 172.21.15.66:0/3991597129 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3376113122 0 0) 0x7f032013b460 con 0x7f03201430c0 2024-03-25T18:11:03.409 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.407+0000 7f031cff9640 1 -- 172.21.15.66:0/3991597129 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f032012d970 con 0x7f03201430c0 2024-03-25T18:11:03.409 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.407+0000 7f031cff9640 1 -- 172.21.15.66:0/3991597129 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f0318001a70 con 0x7f03201430c0 2024-03-25T18:11:03.409 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.407+0000 7f031cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:03.409 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.407+0000 7f031cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:03.409 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.407+0000 7f031cff9640 10 monclient(hunting): dump: 2024-03-25T18:11:03.410 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:03.410 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:03.410 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:03.410 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:03.410 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:03.410 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:03.410 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:03.410 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:03.410 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:03.410 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:03.410 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.407+0000 7f031cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:03.410 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.408+0000 7f031cff9640 1 -- 172.21.15.66:0/3991597129 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3978902679 0 0) 0x7f0300004eb0 con 0x7f03200a98f0 2024-03-25T18:11:03.410 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.408+0000 7f031cff9640 1 -- 172.21.15.66:0/3991597129 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f032013b460 con 0x7f03200a98f0 2024-03-25T18:11:03.410 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.408+0000 7f031cff9640 1 -- 172.21.15.66:0/3991597129 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f0310001be0 con 0x7f03200a98f0 2024-03-25T18:11:03.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.408+0000 7f031cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:03.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.408+0000 7f031cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:11:03.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.408+0000 7f031cff9640 10 monclient(hunting): dump: 2024-03-25T18:11:03.411 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:03.411 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:03.411 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:03.411 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:03.411 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:03.411 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:03.411 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:03.411 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:03.411 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:03.411 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:03.420 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.418+0000 7f031cff9640 1 -- 172.21.15.66:0/3991597129 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1334279155 0 0) 0x7f032012d970 con 0x7f03201430c0 2024-03-25T18:11:03.420 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.418+0000 7f031cff9640 1 -- 172.21.15.66:0/3991597129 >> v1:172.21.15.66:6790/0 conn(0x7f03200a98f0 legacy=0x7f0320139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:03.420 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.418+0000 7f031cff9640 1 -- 172.21.15.66:0/3991597129 >> v1:172.21.15.66:6789/0 conn(0x7f032013f710 legacy=0x7f032012cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:03.420 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.418+0000 7f031cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:11:03.420 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.418+0000 7f031cff9640 1 monclient: found mon.b 2024-03-25T18:11:03.420 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.418+0000 7f031cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:03.420 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.418+0000 7f031cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:11:03.420 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.418+0000 7f031cff9640 1 -- 172.21.15.66:0/3991597129 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f032013b740 con 0x7f03201430c0 2024-03-25T18:11:03.420 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.418+0000 7f031cff9640 10 monclient: _finish_auth 0 2024-03-25T18:11:03.421 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.418+0000 7f031cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:11:03.421 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.418+0000 7f031cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:03.421 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.418+0000 7f032591c640 5 monclient: authenticate success, global_id 4540 2024-03-25T18:11:03.421 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.418+0000 7f032591c640 10 monclient: _renew_subs 2024-03-25T18:11:03.421 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.418+0000 7f032591c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:11:03.421 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.418+0000 7f032591c640 1 -- 172.21.15.66:0/3991597129 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f032012dd80 con 0x7f03201430c0 2024-03-25T18:11:03.421 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.418+0000 7f032591c640 10 client.4540.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:03.421 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.418+0000 7f032591c640 10 monclient: _renew_subs 2024-03-25T18:11:03.421 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.418+0000 7f032591c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:11:03.421 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.418+0000 7f032591c640 1 -- 172.21.15.66:0/3991597129 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f032013a430 con 0x7f03201430c0 2024-03-25T18:11:03.421 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.418+0000 7f032591c640 1 librados: init done 2024-03-25T18:11:03.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.419+0000 7f032591c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:03.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.419+0000 7f032591c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:03.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.419+0000 7f032591c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:11:03.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.419+0000 7f032591c640 1 -- 172.21.15.66:0/3991597129 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f032012dbd0 con 0x7f03201430c0 2024-03-25T18:11:03.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.694+0000 7f031cff9640 1 -- 172.21.15.66:0/3991597129 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0318001ea0 con 0x7f03201430c0 2024-03-25T18:11:03.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.694+0000 7f031cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:03.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.694+0000 7f031cff9640 1 -- 172.21.15.66:0/3991597129 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f0318004d00 con 0x7f03201430c0 2024-03-25T18:11:03.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.694+0000 7f031cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:03.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.694+0000 7f031cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:03.696 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.694+0000 7f031cff9640 10 monclient: dump: 2024-03-25T18:11:03.696 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:03.696 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:03.696 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:03.696 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:03.697 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:03.697 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:03.697 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:03.697 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:03.697 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:03.697 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:03.697 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.694+0000 7f031dffb640 4 set_mon_vals no callback set 2024-03-25T18:11:03.698 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.695+0000 7f031cff9640 1 -- 172.21.15.66:0/3991597129 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f0318004f20 con 0x7f03201430c0 2024-03-25T18:11:03.698 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.696+0000 7f031effd640 10 monclient: get_auth_request con 0x7f030006df10 auth_method 0 2024-03-25T18:11:03.698 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.696+0000 7f031cff9640 10 client.4540.objecter ms_handle_connect 0x7f030006df10 2024-03-25T18:11:03.702 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.701+0000 7f031cff9640 1 -- 172.21.15.66:0/3991597129 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f0318054890 con 0x7f03201430c0 2024-03-25T18:11:03.702 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.701+0000 7f031cff9640 10 client.4540.objecter ms_dispatch 0x7f0320000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:03.703 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.701+0000 7f031cff9640 3 client.4540.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:03.703 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.701+0000 7f031cff9640 3 client.4540.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:03.703 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.701+0000 7f031cff9640 20 client.4540.objecter dump_active .. 0 homeless 2024-03-25T18:11:03.703 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.701+0000 7f031cff9640 1 -- 172.21.15.66:0/3991597129 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f03180503a0 con 0x7f03201430c0 2024-03-25T18:11:03.703 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.701+0000 7f031cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:03.703 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.701+0000 7f031cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:03.937 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.935+0000 7f032591c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:03.937 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.935+0000 7f032591c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:03.937 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.935+0000 7f032591c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:11:03.937 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.935+0000 7f032591c640 1 -- 172.21.15.66:0/3991597129 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03200627b0 con 0x7f03201430c0 2024-03-25T18:11:03.963 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.961+0000 7f031cff9640 1 -- 172.21.15.66:0/3991597129 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f03180522d0 con 0x7f03201430c0 2024-03-25T18:11:03.963 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.961+0000 7f031cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:03.963 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.961+0000 7f031cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:03.963 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:11:03.963 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:03.966 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.964+0000 7f032591c640 10 librados: watch_flush enter 2024-03-25T18:11:03.966 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.964+0000 7f032591c640 10 librados: watch_flush exit 2024-03-25T18:11:03.966 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.964+0000 7f032591c640 20 client.4540.objecter shutdown clearing up homeless session... 2024-03-25T18:11:03.966 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.964+0000 7f032591c640 10 client.4540.objecter successfully canceled tick 2024-03-25T18:11:03.966 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.964+0000 7f032591c640 1 -- 172.21.15.66:0/3991597129 >> v1:172.21.15.193:6800/2275676803 conn(0x7f030006df10 legacy=0x7f03000703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:03.966 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.964+0000 7f032591c640 10 monclient: shutdown 2024-03-25T18:11:03.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.964+0000 7f032591c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:03.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.964+0000 7f032591c640 1 -- 172.21.15.66:0/3991597129 >> v1:172.21.15.193:6789/0 conn(0x7f03201430c0 legacy=0x7f032012d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:03.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.964+0000 7f032591c640 1 -- 172.21.15.66:0/3991597129 shutdown_connections 2024-03-25T18:11:03.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.964+0000 7f032591c640 1 -- 172.21.15.66:0/3991597129 >> 172.21.15.66:0/3991597129 conn(0x7f03200b7ea0 msgr2=0x7f032013d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:03.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.964+0000 7f032591c640 1 -- 172.21.15.66:0/3991597129 shutdown_connections 2024-03-25T18:11:03.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.964+0000 7f032591c640 1 -- 172.21.15.66:0/3991597129 wait complete. 2024-03-25T18:11:03.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:03.964+0000 7f032591c640 1 librados: shutdown 2024-03-25T18:11:03.979 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:09.868 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:07:40.849 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:09.887 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:07:40.868 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:09.921 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:07:40.901 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:09.934 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:07:40.914 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:09.943 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:07:40.924 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:09.947 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:07:40.906 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:09.957 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:07:40.915 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:09.972 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:07:40.952 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:09.974 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:07:40.933 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:10.786 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:10.902 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:10.900+0000 7fe3d8640640 10 monclient: get_monmap_and_config 2024-03-25T18:11:10.902 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:10.900+0000 7fe3d8640640 10 monclient: build_initial_monmap 2024-03-25T18:11:10.902 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:10.900+0000 7fe3d8640640 10 monclient: monmap: 2024-03-25T18:11:10.902 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:11:10.902 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:10.902 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:11:10.902401+0000 2024-03-25T18:11:10.902 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:11:10.902401+0000 2024-03-25T18:11:10.902 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:10.902 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:10.903 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:11:10.903 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:11:10.903 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:11:10.903 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:10.904 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:10.901+0000 7fe3d8640640 1 Processor -- start 2024-03-25T18:11:10.904 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:10.901+0000 7fe3d8640640 1 -- start start 2024-03-25T18:11:10.904 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:10.901+0000 7fe3d8640640 10 monclient: init 2024-03-25T18:11:10.904 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:10.902+0000 7fe3d8640640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:10.904 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:10.902+0000 7fe3d8640640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:11:10.904 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:10.902+0000 7fe3d8640640 10 monclient(hunting): picked mon.noname-a con 0x7fe3d00a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:11:10.904 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:10.902+0000 7fe3d8640640 10 monclient(hunting): picked mon.noname-c con 0x7fe3d013f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:11:10.905 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:10.902+0000 7fe3d8640640 10 monclient(hunting): picked mon.noname-b con 0x7fe3d01430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:11:10.905 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:10.902+0000 7fe3d8640640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3d0142520 con 0x7fe3d00a98f0 2024-03-25T18:11:10.905 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:10.902+0000 7fe3d8640640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3d00b28a0 con 0x7fe3d013f710 2024-03-25T18:11:10.905 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:10.902+0000 7fe3d8640640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3d00b2a30 con 0x7fe3d01430c0 2024-03-25T18:11:10.905 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:10.902+0000 7fe3d8640640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:10.905 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:10.902+0000 7fe3d8640640 10 monclient(hunting): authenticate will time out at 7419.962499s 2024-03-25T18:11:10.905 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:10.902+0000 7fe3d5bdb640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fe3d013f710 0x7fe3d01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:54412/0 (socket says 172.21.15.66:54412) 2024-03-25T18:11:10.905 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:10.902+0000 7fe3d6bdd640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7fe3d01430c0 0x7fe3d014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:46190/0 (socket says 172.21.15.66:46190) 2024-03-25T18:11:10.905 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:10.902+0000 7fe3d6bdd640 1 -- 172.21.15.66:0/3213523730 learned_addr learned my addr 172.21.15.66:0/3213523730 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:11:10.996 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:10.995+0000 7fe3d53da640 1 -- 172.21.15.66:0/3213523730 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3085481380 0 0) 0x7fe3d00b2a30 con 0x7fe3d01430c0 2024-03-25T18:11:10.997 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:10.995+0000 7fe3d53da640 10 monclient(hunting): _negotiate 2024-03-25T18:11:10.997 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:10.995+0000 7fe3d53da640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:10.997 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:10.995+0000 7fe3d53da640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:10.997 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:10.995+0000 7fe3d53da640 10 monclient(hunting): my global_id is 4544 2024-03-25T18:11:10.997 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:10.995+0000 7fe3d53da640 1 -- 172.21.15.66:0/3213523730 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3bc003520 con 0x7fe3d01430c0 2024-03-25T18:11:11.004 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.003+0000 7fe3d53da640 1 -- 172.21.15.66:0/3213523730 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334106049 0 0) 0x7fe3d0142520 con 0x7fe3d00a98f0 2024-03-25T18:11:11.004 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.003+0000 7fe3d53da640 10 monclient(hunting): _negotiate 2024-03-25T18:11:11.004 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.003+0000 7fe3d53da640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:11.004 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.003+0000 7fe3d53da640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:11.004 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.003+0000 7fe3d53da640 10 monclient(hunting): my global_id is 4548 2024-03-25T18:11:11.005 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.003+0000 7fe3d53da640 1 -- 172.21.15.66:0/3213523730 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3d00b2a30 con 0x7fe3d00a98f0 2024-03-25T18:11:11.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.021+0000 7fe3d53da640 1 -- 172.21.15.66:0/3213523730 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2313374854 0 0) 0x7fe3bc003520 con 0x7fe3d01430c0 2024-03-25T18:11:11.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.021+0000 7fe3d53da640 1 -- 172.21.15.66:0/3213523730 >> v1:172.21.15.193:6789/0 conn(0x7fe3d013f710 legacy=0x7fe3d01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:11.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.021+0000 7fe3d53da640 1 -- 172.21.15.66:0/3213523730 >> v1:172.21.15.66:6789/0 conn(0x7fe3d00a98f0 legacy=0x7fe3d00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:11.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.021+0000 7fe3d53da640 10 monclient: _finish_hunting 0 2024-03-25T18:11:11.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.021+0000 7fe3d53da640 1 monclient: found mon.noname-b 2024-03-25T18:11:11.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.021+0000 7fe3d53da640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:11.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.021+0000 7fe3d53da640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.66:6790/0 2024-03-25T18:11:11.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.021+0000 7fe3d53da640 1 -- 172.21.15.66:0/3213523730 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d00b44c0 con 0x7fe3d01430c0 2024-03-25T18:11:11.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.021+0000 7fe3d53da640 10 monclient: _finish_auth 0 2024-03-25T18:11:11.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.021+0000 7fe3d53da640 10 monclient: _check_auth_tickets 2024-03-25T18:11:11.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.021+0000 7fe3d53da640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:11.024 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.021+0000 7fe3d53da640 1 -- 172.21.15.66:0/3213523730 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fe3cc001a10 con 0x7fe3d01430c0 2024-03-25T18:11:11.024 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.021+0000 7fe3d53da640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:11.024 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.021+0000 7fe3d53da640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:11.024 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.021+0000 7fe3d53da640 10 monclient: dump: 2024-03-25T18:11:11.024 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:11.024 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:11.024 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:11.024 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:11.024 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:11.024 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:11.024 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:11.024 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:11.024 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:11.024 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:11.024 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.021+0000 7fe3d8640640 5 monclient: authenticate success, global_id 4544 2024-03-25T18:11:11.024 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.021+0000 7fe3d8640640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:11.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.039+0000 7fe3d53da640 1 -- 172.21.15.66:0/3213523730 <== mon.2 v1:172.21.15.66:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3cc001b70 con 0x7fe3d01430c0 2024-03-25T18:11:11.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.039+0000 7fe3d53da640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:11.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.039+0000 7fe3d53da640 1 -- 172.21.15.66:0/3213523730 <== mon.2 v1:172.21.15.66:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fe3cc001f80 con 0x7fe3d01430c0 2024-03-25T18:11:11.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.039+0000 7fe3d8640640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:11.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.039+0000 7fe3d8640640 4 set_mon_vals no callback set 2024-03-25T18:11:11.042 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.039+0000 7fe3d53da640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:11.042 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.039+0000 7fe3d53da640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:11:11.042 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.039+0000 7fe3d53da640 10 monclient: dump: 2024-03-25T18:11:11.042 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:11.042 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:11.042 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:11.042 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:11.042 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:11.042 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:11.042 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:11.042 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:11.042 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:11.042 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:11.044 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.040+0000 7fe3d8640640 10 monclient: shutdown 2024-03-25T18:11:11.045 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.040+0000 7fe3d8640640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:11.045 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.040+0000 7fe3d8640640 1 -- 172.21.15.66:0/3213523730 >> v1:172.21.15.66:6790/0 conn(0x7fe3d01430c0 legacy=0x7fe3d014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:11.045 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.040+0000 7fe3d8640640 1 -- 172.21.15.66:0/3213523730 shutdown_connections 2024-03-25T18:11:11.045 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.040+0000 7fe3d8640640 1 -- 172.21.15.66:0/3213523730 >> 172.21.15.66:0/3213523730 conn(0x7fe3d00b7ea0 msgr2=0x7fe3d00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:11.045 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.040+0000 7fe3d8640640 1 -- 172.21.15.66:0/3213523730 shutdown_connections 2024-03-25T18:11:11.045 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.040+0000 7fe3d8640640 1 -- 172.21.15.66:0/3213523730 wait complete. 2024-03-25T18:11:11.045 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.041+0000 7fe3d8640640 10 monclient: build_initial_monmap 2024-03-25T18:11:11.045 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.041+0000 7fe3d8640640 10 monclient: monmap: 2024-03-25T18:11:11.045 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:11:11.045 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:11.045 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:11:11.045 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:11:11.045 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:11.045 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:11:11.045 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:11:11.046 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:11:11.046 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:11:11.046 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:11.046 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.041+0000 7fe3d8640640 1 librados: starting msgr at 2024-03-25T18:11:11.046 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.041+0000 7fe3d8640640 1 librados: starting objecter 2024-03-25T18:11:11.046 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.042+0000 7fe3d8640640 1 Processor -- start 2024-03-25T18:11:11.046 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.042+0000 7fe3d8640640 1 -- start start 2024-03-25T18:11:11.046 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.042+0000 7fe3d8640640 1 librados: setting wanted keys 2024-03-25T18:11:11.046 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.042+0000 7fe3d8640640 1 librados: calling monclient init 2024-03-25T18:11:11.046 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.042+0000 7fe3d8640640 10 monclient: init 2024-03-25T18:11:11.046 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.042+0000 7fe3d8640640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:11.046 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.042+0000 7fe3d8640640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:11:11.046 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.042+0000 7fe3d8640640 10 monclient(hunting): picked mon.noname-b con 0x7fe3d00a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:11:11.046 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.042+0000 7fe3d8640640 10 monclient(hunting): picked mon.noname-a con 0x7fe3d013f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:11:11.047 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.042+0000 7fe3d8640640 10 monclient(hunting): picked mon.noname-c con 0x7fe3d01430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:11:11.047 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.042+0000 7fe3d8640640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3d013c2b0 con 0x7fe3d013f710 2024-03-25T18:11:11.047 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.043+0000 7fe3d8640640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3d0136f10 con 0x7fe3d00a98f0 2024-03-25T18:11:11.047 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.043+0000 7fe3d8640640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3d01370a0 con 0x7fe3d01430c0 2024-03-25T18:11:11.047 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.043+0000 7fe3d8640640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:11.047 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.043+0000 7fe3d8640640 10 monclient(hunting): authenticate will time out at 7420.103178s 2024-03-25T18:11:11.047 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.044+0000 7fe3d63dc640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fe3d00a98f0 0x7fe3d01d7590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:54476/0 (socket says 172.21.15.66:54476) 2024-03-25T18:11:11.048 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.044+0000 7fe3d63dc640 1 -- 172.21.15.66:0/2145424255 learned_addr learned my addr 172.21.15.66:0/2145424255 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:11:11.048 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.045+0000 7fe3c6ffd640 10 client.?.objecter ms_handle_connect 0x7fe3d00a98f0 2024-03-25T18:11:11.048 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.045+0000 7fe3c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:11.061 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.059+0000 7fe3c6ffd640 10 client.?.objecter ms_handle_connect 0x7fe3d01430c0 2024-03-25T18:11:11.061 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.059+0000 7fe3c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:11.106 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.105+0000 7fe3c6ffd640 10 client.?.objecter ms_handle_connect 0x7fe3d013f710 2024-03-25T18:11:11.106 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.105+0000 7fe3c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:11.121 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.119+0000 7fe3c6ffd640 1 -- 172.21.15.66:0/2145424255 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060707651 0 0) 0x7fe3d013c2b0 con 0x7fe3d013f710 2024-03-25T18:11:11.121 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.119+0000 7fe3c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:11.121 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.120+0000 7fe3c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:11.121 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.120+0000 7fe3c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:11.121 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.120+0000 7fe3c6ffd640 10 monclient(hunting): my global_id is 4551 2024-03-25T18:11:11.121 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.120+0000 7fe3c6ffd640 1 -- 172.21.15.66:0/2145424255 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3b4004eb0 con 0x7fe3d013f710 2024-03-25T18:11:11.143 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.141+0000 7fe3c6ffd640 1 -- 172.21.15.66:0/2145424255 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1026612774 0 0) 0x7fe3b4004eb0 con 0x7fe3d013f710 2024-03-25T18:11:11.143 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.141+0000 7fe3c6ffd640 1 -- 172.21.15.66:0/2145424255 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3d013c2b0 con 0x7fe3d013f710 2024-03-25T18:11:11.143 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.141+0000 7fe3c6ffd640 1 -- 172.21.15.66:0/2145424255 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fe3c00018c0 con 0x7fe3d013f710 2024-03-25T18:11:11.143 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.141+0000 7fe3c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:11.143 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.142+0000 7fe3c6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:11.143 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.142+0000 7fe3c6ffd640 10 monclient(hunting): dump: 2024-03-25T18:11:11.143 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:11.143 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:11.143 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:11.144 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:11.144 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:11.144 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:11.144 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:11.144 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:11.144 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:11.144 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:11.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.142+0000 7fe3c6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:11.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.154+0000 7fe3c6ffd640 1 -- 172.21.15.66:0/2145424255 <== mon.0 v1:172.21.15.66:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3426348467 0 0) 0x7fe3d013c2b0 con 0x7fe3d013f710 2024-03-25T18:11:11.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.154+0000 7fe3c6ffd640 1 -- 172.21.15.66:0/2145424255 >> v1:172.21.15.66:6790/0 conn(0x7fe3d01430c0 legacy=0x7fe3d0133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:11.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.154+0000 7fe3c6ffd640 1 -- 172.21.15.66:0/2145424255 >> v1:172.21.15.193:6789/0 conn(0x7fe3d00a98f0 legacy=0x7fe3d01d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:11.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.154+0000 7fe3c6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:11:11.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.154+0000 7fe3c6ffd640 1 monclient: found mon.a 2024-03-25T18:11:11.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.154+0000 7fe3c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:11.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.154+0000 7fe3c6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:11:11.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.154+0000 7fe3c6ffd640 1 -- 172.21.15.66:0/2145424255 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d0137230 con 0x7fe3d013f710 2024-03-25T18:11:11.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.154+0000 7fe3c6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:11:11.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.154+0000 7fe3c6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:11:11.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.154+0000 7fe3c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:11.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.154+0000 7fe3d8640640 5 monclient: authenticate success, global_id 4551 2024-03-25T18:11:11.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.154+0000 7fe3d8640640 10 monclient: _renew_subs 2024-03-25T18:11:11.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.154+0000 7fe3d8640640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:11:11.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.154+0000 7fe3d8640640 1 -- 172.21.15.66:0/2145424255 --> v1:172.21.15.66:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3d013b620 con 0x7fe3d013f710 2024-03-25T18:11:11.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.154+0000 7fe3d8640640 10 client.4551.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:11.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.154+0000 7fe3d8640640 10 monclient: _renew_subs 2024-03-25T18:11:11.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.154+0000 7fe3d8640640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:11:11.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.154+0000 7fe3d8640640 1 -- 172.21.15.66:0/2145424255 --> v1:172.21.15.66:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d01338f0 con 0x7fe3d013f710 2024-03-25T18:11:11.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.154+0000 7fe3d8640640 1 librados: init done 2024-03-25T18:11:11.158 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.155+0000 7fe3d8640640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:11.158 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.155+0000 7fe3d8640640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:11.158 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.155+0000 7fe3d8640640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:11:11.158 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.155+0000 7fe3d8640640 1 -- 172.21.15.66:0/2145424255 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3d013b9c0 con 0x7fe3d013f710 2024-03-25T18:11:11.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.477+0000 7fe3c6ffd640 1 -- 172.21.15.66:0/2145424255 <== mon.0 v1:172.21.15.66:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3c00047e0 con 0x7fe3d013f710 2024-03-25T18:11:11.479 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.477+0000 7fe3c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:11.479 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.477+0000 7fe3c6ffd640 1 -- 172.21.15.66:0/2145424255 <== mon.0 v1:172.21.15.66:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fe3c0004a10 con 0x7fe3d013f710 2024-03-25T18:11:11.479 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.477+0000 7fe3c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:11.479 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.477+0000 7fe3c6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:11.479 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.477+0000 7fe3c6ffd640 10 monclient: dump: 2024-03-25T18:11:11.479 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:11.479 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:11.479 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:11.479 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:11.479 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:11.479 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:11.479 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:11.479 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:11.479 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:11.479 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:11.480 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.477+0000 7fe3d4bd9640 4 set_mon_vals no callback set 2024-03-25T18:11:11.480 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.478+0000 7fe3c6ffd640 1 -- 172.21.15.66:0/2145424255 <== mon.0 v1:172.21.15.66:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7fe3c0004c30 con 0x7fe3d013f710 2024-03-25T18:11:11.480 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.479+0000 7fe3d63dc640 10 monclient: get_auth_request con 0x7fe3b406dce0 auth_method 0 2024-03-25T18:11:11.480 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.479+0000 7fe3c6ffd640 10 client.4551.objecter ms_handle_connect 0x7fe3b406dce0 2024-03-25T18:11:11.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.485+0000 7fe3c6ffd640 1 -- 172.21.15.66:0/2145424255 <== mon.0 v1:172.21.15.66:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7fe3c0054610 con 0x7fe3d013f710 2024-03-25T18:11:11.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.485+0000 7fe3c6ffd640 10 client.4551.objecter ms_dispatch 0x7fe3d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:11.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.485+0000 7fe3c6ffd640 3 client.4551.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:11.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.485+0000 7fe3c6ffd640 3 client.4551.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:11.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.485+0000 7fe3c6ffd640 20 client.4551.objecter dump_active .. 0 homeless 2024-03-25T18:11:11.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.485+0000 7fe3c6ffd640 1 -- 172.21.15.66:0/2145424255 <== mon.0 v1:172.21.15.66:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe3c0050120 con 0x7fe3d013f710 2024-03-25T18:11:11.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.485+0000 7fe3c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:11.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.485+0000 7fe3c6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:11.748 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.746+0000 7fe3d8640640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:11.748 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.746+0000 7fe3d8640640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:11.748 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.746+0000 7fe3d8640640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:11:11.748 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.746+0000 7fe3d8640640 1 -- 172.21.15.66:0/2145424255 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3d00627b0 con 0x7fe3d013f710 2024-03-25T18:11:11.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.796+0000 7fe3c6ffd640 1 -- 172.21.15.66:0/2145424255 <== mon.0 v1:172.21.15.66:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7fe3c0050120 con 0x7fe3d013f710 2024-03-25T18:11:11.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.797+0000 7fe3c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:11.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.797+0000 7fe3c6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:11.799 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:11:11.799 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:11.801 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.799+0000 7fe3d8640640 10 librados: watch_flush enter 2024-03-25T18:11:11.801 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.799+0000 7fe3d8640640 10 librados: watch_flush exit 2024-03-25T18:11:11.801 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.799+0000 7fe3d8640640 20 client.4551.objecter shutdown clearing up homeless session... 2024-03-25T18:11:11.801 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.799+0000 7fe3d8640640 10 client.4551.objecter successfully canceled tick 2024-03-25T18:11:11.801 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.799+0000 7fe3d8640640 1 -- 172.21.15.66:0/2145424255 >> v1:172.21.15.193:6800/2275676803 conn(0x7fe3b406dce0 legacy=0x7fe3b40701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:11.801 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.799+0000 7fe3d8640640 10 monclient: shutdown 2024-03-25T18:11:11.801 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.799+0000 7fe3d8640640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:11.801 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.799+0000 7fe3d8640640 1 -- 172.21.15.66:0/2145424255 >> v1:172.21.15.66:6789/0 conn(0x7fe3d013f710 legacy=0x7fe3d01377d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:11.801 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.799+0000 7fe3d8640640 1 -- 172.21.15.66:0/2145424255 shutdown_connections 2024-03-25T18:11:11.801 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.799+0000 7fe3d8640640 1 -- 172.21.15.66:0/2145424255 >> 172.21.15.66:0/2145424255 conn(0x7fe3d00b7ea0 msgr2=0x7fe3d0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:11.801 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.799+0000 7fe3d8640640 1 -- 172.21.15.66:0/2145424255 shutdown_connections 2024-03-25T18:11:11.801 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.799+0000 7fe3d8640640 1 -- 172.21.15.66:0/2145424255 wait complete. 2024-03-25T18:11:11.802 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:11.799+0000 7fe3d8640640 1 librados: shutdown 2024-03-25T18:11:11.812 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:18.619 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:18.721 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.719+0000 7f63e837b640 10 monclient: get_monmap_and_config 2024-03-25T18:11:18.721 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.719+0000 7f63e837b640 10 monclient: build_initial_monmap 2024-03-25T18:11:18.721 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.719+0000 7f63e837b640 10 monclient: monmap: 2024-03-25T18:11:18.721 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:11:18.721 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:18.721 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:11:18.720974+0000 2024-03-25T18:11:18.721 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:11:18.720974+0000 2024-03-25T18:11:18.721 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:18.721 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:18.721 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:11:18.722 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:11:18.722 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:11:18.722 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:18.723 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.720+0000 7f63e837b640 1 Processor -- start 2024-03-25T18:11:18.723 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.720+0000 7f63e837b640 1 -- start start 2024-03-25T18:11:18.723 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.720+0000 7f63e837b640 10 monclient: init 2024-03-25T18:11:18.723 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.720+0000 7f63e837b640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:18.723 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.720+0000 7f63e837b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:11:18.723 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.720+0000 7f63e837b640 10 monclient(hunting): picked mon.noname-b con 0x7f63e00a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:11:18.723 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.720+0000 7f63e837b640 10 monclient(hunting): picked mon.noname-c con 0x7f63e013f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:11:18.723 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.720+0000 7f63e837b640 10 monclient(hunting): picked mon.noname-a con 0x7f63e01430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:11:18.723 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.720+0000 7f63e837b640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63e0142520 con 0x7f63e01430c0 2024-03-25T18:11:18.724 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.721+0000 7f63e837b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63e00b28a0 con 0x7f63e013f710 2024-03-25T18:11:18.724 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.721+0000 7f63e837b640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63e00b2a30 con 0x7f63e00a98f0 2024-03-25T18:11:18.724 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.721+0000 7f63e837b640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:18.724 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.721+0000 7f63e837b640 10 monclient(hunting): authenticate will time out at 7427.781208s 2024-03-25T18:11:18.735 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.734+0000 7f63e5916640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f63e013f710 0x7f63e01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:42392/0 (socket says 172.21.15.66:42392) 2024-03-25T18:11:18.735 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.734+0000 7f63e5916640 1 -- 172.21.15.66:0/3840155030 learned_addr learned my addr 172.21.15.66:0/3840155030 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:11:18.821 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.819+0000 7f63e5115640 1 -- 172.21.15.66:0/3840155030 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180263484 0 0) 0x7f63e0142520 con 0x7f63e01430c0 2024-03-25T18:11:18.821 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.819+0000 7f63e5115640 10 monclient(hunting): _negotiate 2024-03-25T18:11:18.821 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.819+0000 7f63e5115640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:18.821 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.819+0000 7f63e5115640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:18.821 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.819+0000 7f63e5115640 10 monclient(hunting): my global_id is 4554 2024-03-25T18:11:18.821 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.819+0000 7f63e5115640 1 -- 172.21.15.66:0/3840155030 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63cc003520 con 0x7f63e01430c0 2024-03-25T18:11:18.839 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.838+0000 7f63e5115640 1 -- 172.21.15.66:0/3840155030 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4158428409 0 0) 0x7f63cc003520 con 0x7f63e01430c0 2024-03-25T18:11:18.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.838+0000 7f63e5115640 1 -- 172.21.15.66:0/3840155030 >> v1:172.21.15.66:6790/0 conn(0x7f63e00a98f0 legacy=0x7f63e00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:18.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.838+0000 7f63e5115640 1 -- 172.21.15.66:0/3840155030 >> v1:172.21.15.193:6789/0 conn(0x7f63e013f710 legacy=0x7f63e01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:18.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.838+0000 7f63e5115640 10 monclient: _finish_hunting 0 2024-03-25T18:11:18.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.838+0000 7f63e5115640 1 monclient: found mon.noname-a 2024-03-25T18:11:18.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.838+0000 7f63e5115640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:18.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.838+0000 7f63e5115640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.66:6789/0 2024-03-25T18:11:18.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.838+0000 7f63e5115640 1 -- 172.21.15.66:0/3840155030 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e00b44c0 con 0x7f63e01430c0 2024-03-25T18:11:18.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.838+0000 7f63e5115640 10 monclient: _finish_auth 0 2024-03-25T18:11:18.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.838+0000 7f63e5115640 10 monclient: _check_auth_tickets 2024-03-25T18:11:18.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.838+0000 7f63e5115640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:18.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.838+0000 7f63e5115640 1 -- 172.21.15.66:0/3840155030 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f63d8001a10 con 0x7f63e01430c0 2024-03-25T18:11:18.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.838+0000 7f63e5115640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:18.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.838+0000 7f63e5115640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:18.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.838+0000 7f63e5115640 10 monclient: dump: 2024-03-25T18:11:18.840 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:18.840 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:18.841 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:18.841 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:18.841 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:18.841 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:18.841 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:18.841 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:18.841 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:18.841 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:18.841 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.838+0000 7f63e837b640 5 monclient: authenticate success, global_id 4554 2024-03-25T18:11:18.841 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.838+0000 7f63e837b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:18.853 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.851+0000 7f63e6918640 1 -- 172.21.15.66:0/3840155030 >> v1:172.21.15.66:6789/0 conn(0x7f63e01430c0 legacy=0x7f63e014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-25T18:11:18.853 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.851+0000 7f63e6918640 1 -- 172.21.15.66:0/3840155030 >> v1:172.21.15.66:6789/0 conn(0x7f63e01430c0 legacy=0x7f63e014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:11:18.853 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.851+0000 7f63e5115640 1 -- 172.21.15.66:0/3840155030 <== mon.0 v1:172.21.15.66:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63d8001b70 con 0x7f63e01430c0 2024-03-25T18:11:18.853 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.851+0000 7f63e6918640 1 --1- 172.21.15.66:0/3840155030 >> v1:172.21.15.66:6789/0 conn(0x7f63e01430c0 0x7f63e014dc30 :-1 s=OPENED pgs=53602 cs=1 l=1).handle_message read tag failed 2024-03-25T18:11:18.853 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.851+0000 7f63e5115640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:18.853 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.851+0000 7f63e6918640 1 --1- 172.21.15.66:0/3840155030 >> v1:172.21.15.66:6789/0 conn(0x7f63e01430c0 0x7f63e014dc30 :-1 s=OPENED pgs=53602 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:11:18.853 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.851+0000 7f63e837b640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:18.853 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.851+0000 7f63e837b640 4 set_mon_vals no callback set 2024-03-25T18:11:18.853 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.851+0000 7f63e5115640 10 monclient: ms_handle_reset current mon v1:172.21.15.66:6789/0 2024-03-25T18:11:18.853 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.851+0000 7f63e5115640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:18.853 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.851+0000 7f63e5115640 1 -- 172.21.15.66:0/3840155030 >> v1:172.21.15.66:6789/0 conn(0x7f63e01430c0 legacy=0x7f63e014dc30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:11:18.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.852+0000 7f63e837b640 10 monclient: shutdown 2024-03-25T18:11:18.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.852+0000 7f63e5115640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:11:18.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.852+0000 7f63e5115640 10 monclient(hunting): picked mon.b con 0x7f63e013f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:11:18.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.852+0000 7f63e5115640 10 monclient(hunting): picked mon.c con 0x7f63e00a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:11:18.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.852+0000 7f63e5115640 10 monclient(hunting): picked mon.a con 0x7f63cc00b090 addr v1:172.21.15.66:6789/0 2024-03-25T18:11:18.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.852+0000 7f63e5115640 1 -- 172.21.15.66:0/3840155030 --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f63cc003520 con 0x7f63cc00b090 2024-03-25T18:11:18.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.852+0000 7f63e5115640 1 -- 172.21.15.66:0/3840155030 --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f63e0142520 con 0x7f63e013f710 2024-03-25T18:11:18.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.852+0000 7f63e5115640 1 -- 172.21.15.66:0/3840155030 --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f63cc010dc0 con 0x7f63e00a98f0 2024-03-25T18:11:18.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.852+0000 7f63e5115640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:18.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.852+0000 7f63e5115640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.66:6789/0 2024-03-25T18:11:18.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.852+0000 7f63e837b640 20 monclient(hunting): shutdown discarding 1 pending message(s) 2024-03-25T18:11:18.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.852+0000 7f63e837b640 1 -- 172.21.15.66:0/3840155030 >> v1:172.21.15.66:6790/0 conn(0x7f63e00a98f0 legacy=0x7f63cc009970 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-25T18:11:18.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.852+0000 7f63e837b640 1 -- 172.21.15.66:0/3840155030 >> v1:172.21.15.193:6789/0 conn(0x7f63e013f710 legacy=0x7f63cc003af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:11:18.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.852+0000 7f63e837b640 1 -- 172.21.15.66:0/3840155030 >> v1:172.21.15.66:6789/0 conn(0x7f63cc00b090 legacy=0x7f63cc00d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:11:18.858 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.852+0000 7f63e837b640 1 -- 172.21.15.66:0/3840155030 shutdown_connections 2024-03-25T18:11:18.858 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.852+0000 7f63e837b640 1 -- 172.21.15.66:0/3840155030 >> 172.21.15.66:0/3840155030 conn(0x7f63e00b7ea0 msgr2=0x7f63e00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:18.858 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.852+0000 7f63e837b640 1 -- 172.21.15.66:0/3840155030 shutdown_connections 2024-03-25T18:11:18.858 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.852+0000 7f63e837b640 1 -- 172.21.15.66:0/3840155030 wait complete. 2024-03-25T18:11:18.858 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.853+0000 7f63e837b640 10 monclient: build_initial_monmap 2024-03-25T18:11:18.858 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.853+0000 7f63e837b640 10 monclient: monmap: 2024-03-25T18:11:18.858 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:11:18.858 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:18.858 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:11:18.858 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:11:18.858 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:18.858 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:11:18.858 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:11:18.858 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:11:18.859 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:11:18.859 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:18.859 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.853+0000 7f63e837b640 1 librados: starting msgr at 2024-03-25T18:11:18.859 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.853+0000 7f63e837b640 1 librados: starting objecter 2024-03-25T18:11:18.859 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.853+0000 7f63e837b640 1 Processor -- start 2024-03-25T18:11:18.859 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.854+0000 7f63e837b640 1 -- start start 2024-03-25T18:11:18.859 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.854+0000 7f63e837b640 1 librados: setting wanted keys 2024-03-25T18:11:18.859 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.854+0000 7f63e837b640 1 librados: calling monclient init 2024-03-25T18:11:18.859 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.854+0000 7f63e837b640 10 monclient: init 2024-03-25T18:11:18.859 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.854+0000 7f63e837b640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:18.859 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.854+0000 7f63e837b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:11:18.859 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.854+0000 7f63e837b640 10 monclient(hunting): picked mon.noname-c con 0x7f63cc00b090 addr v1:172.21.15.66:6790/0 2024-03-25T18:11:18.859 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.854+0000 7f63e837b640 10 monclient(hunting): picked mon.noname-b con 0x7f63e00a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:11:18.859 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.854+0000 7f63e837b640 10 monclient(hunting): picked mon.noname-a con 0x7f63e013f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:11:18.860 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.854+0000 7f63e837b640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63cc003520 con 0x7f63e013f710 2024-03-25T18:11:18.860 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.854+0000 7f63e837b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63e0142520 con 0x7f63e00a98f0 2024-03-25T18:11:18.860 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.854+0000 7f63e837b640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63cc010dc0 con 0x7f63cc00b090 2024-03-25T18:11:18.860 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.854+0000 7f63e837b640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:18.860 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.854+0000 7f63e837b640 10 monclient(hunting): authenticate will time out at 7427.914722s 2024-03-25T18:11:18.869 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.867+0000 7f63e5916640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f63e00a98f0 0x7f63e0138c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:42432/0 (socket says 172.21.15.66:42432) 2024-03-25T18:11:18.869 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.867+0000 7f63e5916640 1 -- 172.21.15.66:0/2980594895 learned_addr learned my addr 172.21.15.66:0/2980594895 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:11:18.869 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.868+0000 7f63d6ffd640 10 client.?.objecter ms_handle_connect 0x7f63e00a98f0 2024-03-25T18:11:18.869 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.868+0000 7f63d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:18.875 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.873+0000 7f63d6ffd640 10 client.?.objecter ms_handle_connect 0x7f63e013f710 2024-03-25T18:11:18.875 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.873+0000 7f63d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:18.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.880+0000 7f63d6ffd640 1 -- 172.21.15.66:0/2980594895 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1099963257 0 0) 0x7f63e0142520 con 0x7f63e00a98f0 2024-03-25T18:11:18.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.880+0000 7f63d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:18.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.880+0000 7f63d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:18.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.880+0000 7f63d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:18.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.880+0000 7f63d6ffd640 10 monclient(hunting): my global_id is 4552 2024-03-25T18:11:18.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.880+0000 7f63d6ffd640 1 -- 172.21.15.66:0/2980594895 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63c4004eb0 con 0x7f63e00a98f0 2024-03-25T18:11:18.887 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.886+0000 7f63d6ffd640 10 client.?.objecter ms_handle_connect 0x7f63cc00b090 2024-03-25T18:11:18.887 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.886+0000 7f63d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:18.888 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.887+0000 7f63d6ffd640 1 -- 172.21.15.66:0/2980594895 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3589423368 0 0) 0x7f63cc003520 con 0x7f63e013f710 2024-03-25T18:11:18.888 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.887+0000 7f63d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:18.888 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.887+0000 7f63d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:18.889 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.887+0000 7f63d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:18.889 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.887+0000 7f63d6ffd640 10 monclient(hunting): my global_id is 4557 2024-03-25T18:11:18.889 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.887+0000 7f63d6ffd640 1 -- 172.21.15.66:0/2980594895 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63e0142520 con 0x7f63e013f710 2024-03-25T18:11:18.895 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.893+0000 7f63d6ffd640 1 -- 172.21.15.66:0/2980594895 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1388989220 0 0) 0x7f63c4004eb0 con 0x7f63e00a98f0 2024-03-25T18:11:18.895 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.893+0000 7f63d6ffd640 1 -- 172.21.15.66:0/2980594895 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63cc003520 con 0x7f63e00a98f0 2024-03-25T18:11:18.895 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.893+0000 7f63d6ffd640 1 -- 172.21.15.66:0/2980594895 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f63d0001830 con 0x7f63e00a98f0 2024-03-25T18:11:18.895 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.893+0000 7f63d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:18.895 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.893+0000 7f63d6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:18.895 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.893+0000 7f63d6ffd640 10 monclient(hunting): dump: 2024-03-25T18:11:18.896 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:18.896 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:18.896 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:18.896 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:18.896 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:18.896 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:18.896 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:18.896 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:18.896 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:18.896 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:18.897 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.893+0000 7f63d6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:18.908 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.906+0000 7f63d6ffd640 1 -- 172.21.15.66:0/2980594895 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 914181300 0 0) 0x7f63cc003520 con 0x7f63e00a98f0 2024-03-25T18:11:18.908 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.906+0000 7f63d6ffd640 1 -- 172.21.15.66:0/2980594895 >> v1:172.21.15.66:6790/0 conn(0x7f63cc00b090 legacy=0x7f63e01d7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:18.908 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.906+0000 7f63d6ffd640 1 -- 172.21.15.66:0/2980594895 >> v1:172.21.15.66:6789/0 conn(0x7f63e013f710 legacy=0x7f63e0139320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:18.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.906+0000 7f63d6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:11:18.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.906+0000 7f63d6ffd640 1 monclient: found mon.b 2024-03-25T18:11:18.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.906+0000 7f63d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:18.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.906+0000 7f63d6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:11:18.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.906+0000 7f63d6ffd640 1 -- 172.21.15.66:0/2980594895 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63cc0120e0 con 0x7f63e00a98f0 2024-03-25T18:11:18.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.906+0000 7f63d6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:11:18.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.906+0000 7f63d6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:11:18.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.906+0000 7f63d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:18.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.906+0000 7f63e837b640 5 monclient: authenticate success, global_id 4552 2024-03-25T18:11:18.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.906+0000 7f63e837b640 10 monclient: _renew_subs 2024-03-25T18:11:18.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.906+0000 7f63e837b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:11:18.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.906+0000 7f63e837b640 1 -- 172.21.15.66:0/2980594895 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63e01d7720 con 0x7f63e00a98f0 2024-03-25T18:11:18.909 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.906+0000 7f63e837b640 10 client.4552.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:18.910 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.906+0000 7f63e837b640 10 monclient: _renew_subs 2024-03-25T18:11:18.910 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.906+0000 7f63e837b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:11:18.910 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.906+0000 7f63e837b640 1 -- 172.21.15.66:0/2980594895 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63e0139a80 con 0x7f63e00a98f0 2024-03-25T18:11:18.910 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.906+0000 7f63e837b640 1 librados: init done 2024-03-25T18:11:18.910 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.907+0000 7f63e837b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:18.910 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.907+0000 7f63e837b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:18.910 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.907+0000 7f63e837b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:11:18.911 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:18.907+0000 7f63e837b640 1 -- 172.21.15.66:0/2980594895 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63e00627b0 con 0x7f63e00a98f0 2024-03-25T18:11:19.187 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.185+0000 7f63d6ffd640 1 -- 172.21.15.66:0/2980594895 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63d0001d20 con 0x7f63e00a98f0 2024-03-25T18:11:19.187 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.185+0000 7f63d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:19.188 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.185+0000 7f63d6ffd640 1 -- 172.21.15.66:0/2980594895 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f63d00047e0 con 0x7f63e00a98f0 2024-03-25T18:11:19.188 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.185+0000 7f63d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:19.188 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.185+0000 7f63d6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:19.188 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.185+0000 7f63d6ffd640 10 monclient: dump: 2024-03-25T18:11:19.188 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:19.188 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:19.188 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:19.188 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:19.188 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:19.188 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:19.188 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:19.188 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:19.188 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:19.189 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:19.189 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.185+0000 7f63e4914640 4 set_mon_vals no callback set 2024-03-25T18:11:19.189 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.186+0000 7f63d6ffd640 1 -- 172.21.15.66:0/2980594895 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f63d0004a00 con 0x7f63e00a98f0 2024-03-25T18:11:19.189 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.187+0000 7f63e6117640 10 monclient: get_auth_request con 0x7f63c406dc70 auth_method 0 2024-03-25T18:11:19.189 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.188+0000 7f63d6ffd640 10 client.4552.objecter ms_handle_connect 0x7f63c406dc70 2024-03-25T18:11:19.195 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.194+0000 7f63d6ffd640 1 -- 172.21.15.66:0/2980594895 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f63d00545d0 con 0x7f63e00a98f0 2024-03-25T18:11:19.196 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.194+0000 7f63d6ffd640 10 client.4552.objecter ms_dispatch 0x7f63e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:19.196 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.194+0000 7f63d6ffd640 3 client.4552.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:19.196 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.194+0000 7f63d6ffd640 3 client.4552.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:19.196 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.194+0000 7f63d6ffd640 20 client.4552.objecter dump_active .. 0 homeless 2024-03-25T18:11:19.196 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.195+0000 7f63d6ffd640 1 -- 172.21.15.66:0/2980594895 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f63d004ff10 con 0x7f63e00a98f0 2024-03-25T18:11:19.196 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.195+0000 7f63d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:19.196 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.195+0000 7f63d6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:19.464 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.462+0000 7f63e837b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:19.464 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.462+0000 7f63e837b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:19.464 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.462+0000 7f63e837b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:11:19.464 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.462+0000 7f63e837b640 1 -- 172.21.15.66:0/2980594895 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63e0141a00 con 0x7f63e00a98f0 2024-03-25T18:11:19.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.495+0000 7f63d6ffd640 1 -- 172.21.15.66:0/2980594895 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f63d0052010 con 0x7f63e00a98f0 2024-03-25T18:11:19.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.495+0000 7f63d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:19.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.495+0000 7f63d6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:19.497 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:11:19.498 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:19.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.498+0000 7f63e837b640 10 librados: watch_flush enter 2024-03-25T18:11:19.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.498+0000 7f63e837b640 10 librados: watch_flush exit 2024-03-25T18:11:19.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.498+0000 7f63e837b640 20 client.4552.objecter shutdown clearing up homeless session... 2024-03-25T18:11:19.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.498+0000 7f63e837b640 10 client.4552.objecter successfully canceled tick 2024-03-25T18:11:19.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.498+0000 7f63e837b640 1 -- 172.21.15.66:0/2980594895 >> v1:172.21.15.193:6800/2275676803 conn(0x7f63c406dc70 legacy=0x7f63c4070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:19.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.498+0000 7f63e837b640 10 monclient: shutdown 2024-03-25T18:11:19.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.498+0000 7f63e837b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:19.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.498+0000 7f63e837b640 1 -- 172.21.15.66:0/2980594895 >> v1:172.21.15.193:6789/0 conn(0x7f63e00a98f0 legacy=0x7f63e0138c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:19.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.498+0000 7f63e837b640 1 -- 172.21.15.66:0/2980594895 shutdown_connections 2024-03-25T18:11:19.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.498+0000 7f63e837b640 1 -- 172.21.15.66:0/2980594895 >> 172.21.15.66:0/2980594895 conn(0x7f63e00b7ea0 msgr2=0x7f63e0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:19.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.498+0000 7f63e837b640 1 -- 172.21.15.66:0/2980594895 shutdown_connections 2024-03-25T18:11:19.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.498+0000 7f63e837b640 1 -- 172.21.15.66:0/2980594895 wait complete. 2024-03-25T18:11:19.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:19.498+0000 7f63e837b640 1 librados: shutdown 2024-03-25T18:11:19.512 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:26.318 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:26.441 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.440+0000 7fb84ccae640 10 monclient: get_monmap_and_config 2024-03-25T18:11:26.441 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.440+0000 7fb84ccae640 10 monclient: build_initial_monmap 2024-03-25T18:11:26.442 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.440+0000 7fb84ccae640 10 monclient: monmap: 2024-03-25T18:11:26.442 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:11:26.442 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:26.442 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:11:26.441712+0000 2024-03-25T18:11:26.442 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:11:26.441712+0000 2024-03-25T18:11:26.442 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:26.442 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:26.442 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:11:26.442 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:11:26.442 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:11:26.442 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:26.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.441+0000 7fb84ccae640 1 Processor -- start 2024-03-25T18:11:26.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.441+0000 7fb84ccae640 1 -- start start 2024-03-25T18:11:26.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.441+0000 7fb84ccae640 10 monclient: init 2024-03-25T18:11:26.444 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.442+0000 7fb84ccae640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:26.444 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.442+0000 7fb84ccae640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:11:26.444 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.443+0000 7fb84ccae640 10 monclient(hunting): picked mon.noname-a con 0x7fb8480a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:11:26.444 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.443+0000 7fb84ccae640 10 monclient(hunting): picked mon.noname-c con 0x7fb84813f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:11:26.444 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.443+0000 7fb84ccae640 10 monclient(hunting): picked mon.noname-b con 0x7fb8481430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:11:26.444 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.443+0000 7fb84ccae640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb848142520 con 0x7fb8480a98f0 2024-03-25T18:11:26.445 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.443+0000 7fb84ccae640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8480b28a0 con 0x7fb84813f710 2024-03-25T18:11:26.445 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.443+0000 7fb84ccae640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8480b2a30 con 0x7fb8481430c0 2024-03-25T18:11:26.445 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.443+0000 7fb84ccae640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:26.445 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.443+0000 7fb84ccae640 10 monclient(hunting): authenticate will time out at 7435.503199s 2024-03-25T18:11:26.460 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.459+0000 7fb84659c640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7fb8480a98f0 0x7fb8480a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:49186/0 (socket says 172.21.15.66:49186) 2024-03-25T18:11:26.460 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.459+0000 7fb84659c640 1 -- 172.21.15.66:0/2463977462 learned_addr learned my addr 172.21.15.66:0/2463977462 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:11:26.481 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.480+0000 7fb84559a640 1 -- 172.21.15.66:0/2463977462 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2752092720 0 0) 0x7fb848142520 con 0x7fb8480a98f0 2024-03-25T18:11:26.482 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.480+0000 7fb84559a640 10 monclient(hunting): _negotiate 2024-03-25T18:11:26.482 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.480+0000 7fb84559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:26.482 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.480+0000 7fb84559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:26.482 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.480+0000 7fb84559a640 10 monclient(hunting): my global_id is 4560 2024-03-25T18:11:26.482 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.480+0000 7fb84559a640 1 -- 172.21.15.66:0/2463977462 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb830003520 con 0x7fb8480a98f0 2024-03-25T18:11:26.485 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.483+0000 7fb84559a640 1 -- 172.21.15.66:0/2463977462 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024413559 0 0) 0x7fb8480b28a0 con 0x7fb84813f710 2024-03-25T18:11:26.485 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.483+0000 7fb84559a640 10 monclient(hunting): _negotiate 2024-03-25T18:11:26.485 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.483+0000 7fb84559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:26.485 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.483+0000 7fb84559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:26.485 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.483+0000 7fb84559a640 10 monclient(hunting): my global_id is 4555 2024-03-25T18:11:26.485 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.483+0000 7fb84559a640 1 -- 172.21.15.66:0/2463977462 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb848142520 con 0x7fb84813f710 2024-03-25T18:11:26.494 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.492+0000 7fb84559a640 1 -- 172.21.15.66:0/2463977462 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034158936 0 0) 0x7fb8480b2a30 con 0x7fb8481430c0 2024-03-25T18:11:26.494 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.492+0000 7fb84559a640 10 monclient(hunting): _negotiate 2024-03-25T18:11:26.494 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.492+0000 7fb84559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:26.494 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.492+0000 7fb84559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:26.494 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.492+0000 7fb84559a640 10 monclient(hunting): my global_id is 4556 2024-03-25T18:11:26.494 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.492+0000 7fb84559a640 1 -- 172.21.15.66:0/2463977462 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8480b28a0 con 0x7fb8481430c0 2024-03-25T18:11:26.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.494+0000 7fb84559a640 1 -- 172.21.15.66:0/2463977462 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1368667785 0 0) 0x7fb830003520 con 0x7fb8480a98f0 2024-03-25T18:11:26.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.494+0000 7fb84559a640 1 -- 172.21.15.66:0/2463977462 >> v1:172.21.15.66:6790/0 conn(0x7fb8481430c0 legacy=0x7fb84814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:26.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.494+0000 7fb84559a640 1 -- 172.21.15.66:0/2463977462 >> v1:172.21.15.193:6789/0 conn(0x7fb84813f710 legacy=0x7fb8481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:26.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.494+0000 7fb84559a640 10 monclient: _finish_hunting 0 2024-03-25T18:11:26.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.494+0000 7fb84559a640 1 monclient: found mon.noname-a 2024-03-25T18:11:26.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.494+0000 7fb84559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:26.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.494+0000 7fb84559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.66:6789/0 2024-03-25T18:11:26.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.494+0000 7fb84559a640 1 -- 172.21.15.66:0/2463977462 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8480b44c0 con 0x7fb8480a98f0 2024-03-25T18:11:26.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.494+0000 7fb84559a640 10 monclient: _finish_auth 0 2024-03-25T18:11:26.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.494+0000 7fb84559a640 10 monclient: _check_auth_tickets 2024-03-25T18:11:26.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.494+0000 7fb84559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:26.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.494+0000 7fb84559a640 1 -- 172.21.15.66:0/2463977462 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fb838001690 con 0x7fb8480a98f0 2024-03-25T18:11:26.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.494+0000 7fb84559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:26.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.494+0000 7fb84559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:26.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.494+0000 7fb84559a640 10 monclient: dump: 2024-03-25T18:11:26.497 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:26.497 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:26.497 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:26.497 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:26.497 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:26.498 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:26.498 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:26.498 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:26.498 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:26.498 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:26.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.495+0000 7fb84ccae640 5 monclient: authenticate success, global_id 4560 2024-03-25T18:11:26.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.495+0000 7fb84ccae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:26.538 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.536+0000 7fb84559a640 1 -- 172.21.15.66:0/2463977462 <== mon.0 v1:172.21.15.66:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8380017f0 con 0x7fb8480a98f0 2024-03-25T18:11:26.538 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.536+0000 7fb84559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:26.538 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.536+0000 7fb84559a640 1 -- 172.21.15.66:0/2463977462 <== mon.0 v1:172.21.15.66:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fb838001c00 con 0x7fb8480a98f0 2024-03-25T18:11:26.538 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.536+0000 7fb84559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:26.538 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.536+0000 7fb84559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:26.538 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.536+0000 7fb84559a640 10 monclient: dump: 2024-03-25T18:11:26.538 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:26.538 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:26.538 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:26.539 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:26.539 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:26.539 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:26.539 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:26.539 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:26.539 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:26.539 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:26.539 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.536+0000 7fb84ccae640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:26.539 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.536+0000 7fb84ccae640 4 set_mon_vals no callback set 2024-03-25T18:11:26.540 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.537+0000 7fb84ccae640 10 monclient: shutdown 2024-03-25T18:11:26.540 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.537+0000 7fb84ccae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:26.540 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.537+0000 7fb84ccae640 1 -- 172.21.15.66:0/2463977462 >> v1:172.21.15.66:6789/0 conn(0x7fb8480a98f0 legacy=0x7fb8480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:26.540 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.537+0000 7fb84ccae640 1 -- 172.21.15.66:0/2463977462 shutdown_connections 2024-03-25T18:11:26.541 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.537+0000 7fb84ccae640 1 -- 172.21.15.66:0/2463977462 >> 172.21.15.66:0/2463977462 conn(0x7fb8480b7ea0 msgr2=0x7fb8480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:26.541 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.537+0000 7fb84ccae640 1 -- 172.21.15.66:0/2463977462 shutdown_connections 2024-03-25T18:11:26.541 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.538+0000 7fb84ccae640 1 -- 172.21.15.66:0/2463977462 wait complete. 2024-03-25T18:11:26.541 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.539+0000 7fb84ccae640 10 monclient: build_initial_monmap 2024-03-25T18:11:26.541 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.539+0000 7fb84ccae640 10 monclient: monmap: 2024-03-25T18:11:26.541 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:11:26.541 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:26.541 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:11:26.541 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:11:26.541 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:26.541 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:11:26.541 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:11:26.541 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:11:26.541 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:11:26.541 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:26.542 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.539+0000 7fb84ccae640 1 librados: starting msgr at 2024-03-25T18:11:26.542 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.539+0000 7fb84ccae640 1 librados: starting objecter 2024-03-25T18:11:26.542 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.539+0000 7fb84ccae640 1 Processor -- start 2024-03-25T18:11:26.543 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.539+0000 7fb84ccae640 1 -- start start 2024-03-25T18:11:26.543 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.539+0000 7fb84ccae640 1 librados: setting wanted keys 2024-03-25T18:11:26.543 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.539+0000 7fb84ccae640 1 librados: calling monclient init 2024-03-25T18:11:26.543 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.539+0000 7fb84ccae640 10 monclient: init 2024-03-25T18:11:26.543 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.540+0000 7fb84ccae640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:26.543 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.540+0000 7fb84ccae640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:11:26.543 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.540+0000 7fb84ccae640 10 monclient(hunting): picked mon.noname-a con 0x7fb8480a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:11:26.543 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.540+0000 7fb84ccae640 10 monclient(hunting): picked mon.noname-b con 0x7fb84813f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:11:26.543 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.540+0000 7fb84ccae640 10 monclient(hunting): picked mon.noname-c con 0x7fb8481430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:11:26.543 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.540+0000 7fb84ccae640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8481db2f0 con 0x7fb8480a98f0 2024-03-25T18:11:26.543 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.540+0000 7fb84ccae640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb848139180 con 0x7fb84813f710 2024-03-25T18:11:26.543 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.540+0000 7fb84ccae640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb848137670 con 0x7fb8481430c0 2024-03-25T18:11:26.544 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.540+0000 7fb84ccae640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:26.544 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.540+0000 7fb84ccae640 10 monclient(hunting): authenticate will time out at 7435.600666s 2024-03-25T18:11:26.550 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.548+0000 7fb845d9b640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fb84813f710 0x7fb848132f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:54146/0 (socket says 172.21.15.66:54146) 2024-03-25T18:11:26.550 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.548+0000 7fb845d9b640 1 -- 172.21.15.66:0/1321021458 learned_addr learned my addr 172.21.15.66:0/1321021458 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:11:26.551 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.549+0000 7fb836ffd640 10 client.?.objecter ms_handle_connect 0x7fb84813f710 2024-03-25T18:11:26.551 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.549+0000 7fb836ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:26.556 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.554+0000 7fb836ffd640 10 client.?.objecter ms_handle_connect 0x7fb8481430c0 2024-03-25T18:11:26.556 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.554+0000 7fb836ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:26.614 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.613+0000 7fb836ffd640 10 client.?.objecter ms_handle_connect 0x7fb8480a98f0 2024-03-25T18:11:26.614 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.613+0000 7fb836ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:26.642 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.641+0000 7fb836ffd640 1 -- 172.21.15.66:0/1321021458 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3911134477 0 0) 0x7fb8481db2f0 con 0x7fb8480a98f0 2024-03-25T18:11:26.642 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.641+0000 7fb836ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:26.643 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.641+0000 7fb836ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:26.643 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.641+0000 7fb836ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:26.643 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.641+0000 7fb836ffd640 10 monclient(hunting): my global_id is 4563 2024-03-25T18:11:26.643 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.641+0000 7fb836ffd640 1 -- 172.21.15.66:0/1321021458 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb828004eb0 con 0x7fb8480a98f0 2024-03-25T18:11:26.670 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.669+0000 7fb836ffd640 1 -- 172.21.15.66:0/1321021458 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 729628341 0 0) 0x7fb828004eb0 con 0x7fb8480a98f0 2024-03-25T18:11:26.670 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.669+0000 7fb836ffd640 1 -- 172.21.15.66:0/1321021458 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8481db2f0 con 0x7fb8480a98f0 2024-03-25T18:11:26.670 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.669+0000 7fb836ffd640 1 -- 172.21.15.66:0/1321021458 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fb838001b00 con 0x7fb8480a98f0 2024-03-25T18:11:26.671 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.669+0000 7fb836ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:26.671 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.669+0000 7fb836ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:26.671 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.669+0000 7fb836ffd640 10 monclient(hunting): dump: 2024-03-25T18:11:26.671 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:26.671 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:26.671 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:26.671 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:26.671 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:26.671 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:26.671 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:26.671 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:26.671 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:26.671 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:26.671 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.669+0000 7fb836ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:26.678 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.677+0000 7fb836ffd640 1 -- 172.21.15.66:0/1321021458 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243403720 0 0) 0x7fb848137670 con 0x7fb8481430c0 2024-03-25T18:11:26.679 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.677+0000 7fb836ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:26.679 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.677+0000 7fb836ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:26.679 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.677+0000 7fb836ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:26.679 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.677+0000 7fb836ffd640 10 monclient(hunting): my global_id is 4559 2024-03-25T18:11:26.679 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.677+0000 7fb836ffd640 1 -- 172.21.15.66:0/1321021458 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb828004eb0 con 0x7fb8481430c0 2024-03-25T18:11:26.686 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.684+0000 7fb836ffd640 1 -- 172.21.15.66:0/1321021458 <== mon.0 v1:172.21.15.66:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 86722707 0 0) 0x7fb8481db2f0 con 0x7fb8480a98f0 2024-03-25T18:11:26.686 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.684+0000 7fb836ffd640 1 -- 172.21.15.66:0/1321021458 >> v1:172.21.15.66:6790/0 conn(0x7fb8481430c0 legacy=0x7fb8481e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:26.687 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.684+0000 7fb836ffd640 1 -- 172.21.15.66:0/1321021458 >> v1:172.21.15.193:6789/0 conn(0x7fb84813f710 legacy=0x7fb848132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:26.687 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.684+0000 7fb836ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:11:26.687 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.684+0000 7fb836ffd640 1 monclient: found mon.a 2024-03-25T18:11:26.687 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.684+0000 7fb836ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:26.687 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.684+0000 7fb836ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:11:26.687 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.684+0000 7fb836ffd640 1 -- 172.21.15.66:0/1321021458 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb848137200 con 0x7fb8480a98f0 2024-03-25T18:11:26.687 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.684+0000 7fb836ffd640 10 monclient: _finish_auth 0 2024-03-25T18:11:26.687 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.684+0000 7fb836ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:11:26.687 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.684+0000 7fb836ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:26.687 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.684+0000 7fb84ccae640 5 monclient: authenticate success, global_id 4563 2024-03-25T18:11:26.687 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.684+0000 7fb84ccae640 10 monclient: _renew_subs 2024-03-25T18:11:26.687 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.684+0000 7fb84ccae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:11:26.687 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.684+0000 7fb84ccae640 1 -- 172.21.15.66:0/1321021458 --> v1:172.21.15.66:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8480b7290 con 0x7fb8480a98f0 2024-03-25T18:11:26.688 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.685+0000 7fb84ccae640 10 client.4563.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:26.688 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.685+0000 7fb84ccae640 10 monclient: _renew_subs 2024-03-25T18:11:26.688 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.685+0000 7fb84ccae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:11:26.688 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.685+0000 7fb84ccae640 1 -- 172.21.15.66:0/1321021458 --> v1:172.21.15.66:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb848133680 con 0x7fb8480a98f0 2024-03-25T18:11:26.688 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.685+0000 7fb84ccae640 1 librados: init done 2024-03-25T18:11:26.688 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.685+0000 7fb84ccae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:26.688 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.686+0000 7fb84ccae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:26.689 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.686+0000 7fb84ccae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:11:26.689 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:26.686+0000 7fb84ccae640 1 -- 172.21.15.66:0/1321021458 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb848137800 con 0x7fb8480a98f0 2024-03-25T18:11:27.021 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.019+0000 7fb836ffd640 1 -- 172.21.15.66:0/1321021458 <== mon.0 v1:172.21.15.66:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb838001220 con 0x7fb8480a98f0 2024-03-25T18:11:27.021 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.019+0000 7fb836ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:27.021 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.019+0000 7fb836ffd640 1 -- 172.21.15.66:0/1321021458 <== mon.0 v1:172.21.15.66:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fb838004c30 con 0x7fb8480a98f0 2024-03-25T18:11:27.021 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.019+0000 7fb836ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:27.021 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.019+0000 7fb836ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:27.021 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.019+0000 7fb836ffd640 10 monclient: dump: 2024-03-25T18:11:27.021 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:27.021 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:27.021 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:27.021 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:27.021 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:27.021 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:27.022 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:27.022 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:27.022 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:27.022 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:27.022 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.019+0000 7fb844d99640 4 set_mon_vals no callback set 2024-03-25T18:11:27.022 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.020+0000 7fb836ffd640 1 -- 172.21.15.66:0/1321021458 <== mon.0 v1:172.21.15.66:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7fb83801ac40 con 0x7fb8480a98f0 2024-03-25T18:11:27.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.021+0000 7fb845d9b640 10 monclient: get_auth_request con 0x7fb82806dd40 auth_method 0 2024-03-25T18:11:27.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.021+0000 7fb836ffd640 10 client.4563.objecter ms_handle_connect 0x7fb82806dd40 2024-03-25T18:11:27.030 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.028+0000 7fb836ffd640 1 -- 172.21.15.66:0/1321021458 <== mon.0 v1:172.21.15.66:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7fb838054800 con 0x7fb8480a98f0 2024-03-25T18:11:27.030 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.028+0000 7fb836ffd640 10 client.4563.objecter ms_dispatch 0x7fb848000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:27.030 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.028+0000 7fb836ffd640 3 client.4563.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:27.030 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.028+0000 7fb836ffd640 3 client.4563.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:27.030 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.028+0000 7fb836ffd640 20 client.4563.objecter dump_active .. 0 homeless 2024-03-25T18:11:27.030 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.028+0000 7fb836ffd640 1 -- 172.21.15.66:0/1321021458 <== mon.0 v1:172.21.15.66:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb838050140 con 0x7fb8480a98f0 2024-03-25T18:11:27.030 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.028+0000 7fb836ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:27.030 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.028+0000 7fb836ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:27.288 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.286+0000 7fb84ccae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:27.288 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.286+0000 7fb84ccae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:27.288 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.286+0000 7fb84ccae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:11:27.288 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.286+0000 7fb84ccae640 1 -- 172.21.15.66:0/1321021458 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8480627b0 con 0x7fb8480a98f0 2024-03-25T18:11:27.333 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.332+0000 7fb836ffd640 1 -- 172.21.15.66:0/1321021458 <== mon.0 v1:172.21.15.66:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7fb838052240 con 0x7fb8480a98f0 2024-03-25T18:11:27.333 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.332+0000 7fb836ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:27.333 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.332+0000 7fb836ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:27.334 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:11:27.334 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:27.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.334+0000 7fb84ccae640 10 librados: watch_flush enter 2024-03-25T18:11:27.336 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.334+0000 7fb84ccae640 10 librados: watch_flush exit 2024-03-25T18:11:27.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.334+0000 7fb84ccae640 20 client.4563.objecter shutdown clearing up homeless session... 2024-03-25T18:11:27.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.334+0000 7fb84ccae640 10 client.4563.objecter successfully canceled tick 2024-03-25T18:11:27.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.334+0000 7fb84ccae640 1 -- 172.21.15.66:0/1321021458 >> v1:172.21.15.193:6800/2275676803 conn(0x7fb82806dd40 legacy=0x7fb828070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:27.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.334+0000 7fb84ccae640 10 monclient: shutdown 2024-03-25T18:11:27.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.334+0000 7fb84ccae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:27.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.334+0000 7fb84ccae640 1 -- 172.21.15.66:0/1321021458 >> v1:172.21.15.66:6789/0 conn(0x7fb8480a98f0 legacy=0x7fb8481dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:27.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.334+0000 7fb84ccae640 1 -- 172.21.15.66:0/1321021458 shutdown_connections 2024-03-25T18:11:27.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.334+0000 7fb84ccae640 1 -- 172.21.15.66:0/1321021458 >> 172.21.15.66:0/1321021458 conn(0x7fb8480b7ea0 msgr2=0x7fb848074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:27.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.334+0000 7fb84ccae640 1 -- 172.21.15.66:0/1321021458 shutdown_connections 2024-03-25T18:11:27.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.334+0000 7fb84ccae640 1 -- 172.21.15.66:0/1321021458 wait complete. 2024-03-25T18:11:27.337 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:27.335+0000 7fb84ccae640 1 librados: shutdown 2024-03-25T18:11:27.347 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:29.954 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:08:00.934 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:29.974 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:08:00.941 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:29.993 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:08:00.974 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:30.005 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:08:00.963 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:30.012 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:08:00.971 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:30.020 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:08:01.001 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:30.026 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:08:01.007 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:30.050 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:08:01.009 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:30.066 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:08:01.046 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:34.154 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:34.269 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.267+0000 7f37f664a640 10 monclient: get_monmap_and_config 2024-03-25T18:11:34.269 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.267+0000 7f37f664a640 10 monclient: build_initial_monmap 2024-03-25T18:11:34.269 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.267+0000 7f37f664a640 10 monclient: monmap: 2024-03-25T18:11:34.269 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:11:34.269 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:34.269 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:11:34.269443+0000 2024-03-25T18:11:34.269 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:11:34.269443+0000 2024-03-25T18:11:34.269 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:34.269 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:34.270 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:11:34.270 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:11:34.270 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:11:34.270 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:34.271 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.268+0000 7f37f664a640 1 Processor -- start 2024-03-25T18:11:34.271 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.268+0000 7f37f664a640 1 -- start start 2024-03-25T18:11:34.271 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.268+0000 7f37f664a640 10 monclient: init 2024-03-25T18:11:34.271 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.269+0000 7f37f664a640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:34.271 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.269+0000 7f37f664a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:11:34.271 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.269+0000 7f37f664a640 10 monclient(hunting): picked mon.noname-c con 0x7f37f00b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T18:11:34.271 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.269+0000 7f37f664a640 10 monclient(hunting): picked mon.noname-a con 0x7f37f00b1730 addr v1:172.21.15.66:6789/0 2024-03-25T18:11:34.271 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.269+0000 7f37f664a640 10 monclient(hunting): picked mon.noname-b con 0x7f37f014be90 addr v1:172.21.15.66:6790/0 2024-03-25T18:11:34.272 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.269+0000 7f37f664a640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f00af940 con 0x7f37f00b1730 2024-03-25T18:11:34.272 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.269+0000 7f37f664a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f00b0a70 con 0x7f37f00b4d60 2024-03-25T18:11:34.272 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.269+0000 7f37f664a640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f00b0d50 con 0x7f37f014be90 2024-03-25T18:11:34.272 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.269+0000 7f37f664a640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:34.272 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.269+0000 7f37f664a640 10 monclient(hunting): authenticate will time out at 7443.330010s 2024-03-25T18:11:34.279 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.278+0000 7f37f4be7640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f37f014be90 0x7f37f014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:42530/0 (socket says 172.21.15.66:42530) 2024-03-25T18:11:34.280 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.278+0000 7f37f4be7640 1 -- 172.21.15.66:0/2279971917 learned_addr learned my addr 172.21.15.66:0/2279971917 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:11:34.288 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.286+0000 7f37eeffd640 1 -- 172.21.15.66:0/2279971917 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688125097 0 0) 0x7f37f00b0d50 con 0x7f37f014be90 2024-03-25T18:11:34.288 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.286+0000 7f37eeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:34.288 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.286+0000 7f37eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:34.288 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.286+0000 7f37eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:34.288 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.286+0000 7f37eeffd640 10 monclient(hunting): my global_id is 4562 2024-03-25T18:11:34.288 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.286+0000 7f37eeffd640 1 -- 172.21.15.66:0/2279971917 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37d8003520 con 0x7f37f014be90 2024-03-25T18:11:34.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.295+0000 7f37eeffd640 1 -- 172.21.15.66:0/2279971917 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3527898148 0 0) 0x7f37f00b0a70 con 0x7f37f00b4d60 2024-03-25T18:11:34.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.295+0000 7f37eeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:34.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.295+0000 7f37eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:34.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.295+0000 7f37eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:34.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.295+0000 7f37eeffd640 10 monclient(hunting): my global_id is 4561 2024-03-25T18:11:34.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.295+0000 7f37eeffd640 1 -- 172.21.15.66:0/2279971917 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37f00b0d50 con 0x7f37f00b4d60 2024-03-25T18:11:34.299 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.297+0000 7f37eeffd640 1 -- 172.21.15.66:0/2279971917 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828670795 0 0) 0x7f37f00af940 con 0x7f37f00b1730 2024-03-25T18:11:34.299 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.297+0000 7f37eeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:34.299 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.297+0000 7f37eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:34.299 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.297+0000 7f37eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:34.299 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.297+0000 7f37eeffd640 10 monclient(hunting): my global_id is 4566 2024-03-25T18:11:34.299 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.297+0000 7f37eeffd640 1 -- 172.21.15.66:0/2279971917 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37f00b0a70 con 0x7f37f00b1730 2024-03-25T18:11:34.304 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.302+0000 7f37eeffd640 1 -- 172.21.15.66:0/2279971917 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3589475291 0 0) 0x7f37f00b0d50 con 0x7f37f00b4d60 2024-03-25T18:11:34.304 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.302+0000 7f37eeffd640 1 -- 172.21.15.66:0/2279971917 >> v1:172.21.15.66:6790/0 conn(0x7f37f014be90 legacy=0x7f37f014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:34.304 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.302+0000 7f37eeffd640 1 -- 172.21.15.66:0/2279971917 >> v1:172.21.15.66:6789/0 conn(0x7f37f00b1730 legacy=0x7f37f00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:34.304 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.302+0000 7f37eeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:11:34.304 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.302+0000 7f37eeffd640 1 monclient: found mon.noname-c 2024-03-25T18:11:34.304 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.302+0000 7f37eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:34.304 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.302+0000 7f37eeffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:11:34.304 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.302+0000 7f37eeffd640 1 -- 172.21.15.66:0/2279971917 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f00b1030 con 0x7f37f00b4d60 2024-03-25T18:11:34.305 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.302+0000 7f37eeffd640 10 monclient: _finish_auth 0 2024-03-25T18:11:34.305 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.302+0000 7f37eeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:11:34.305 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.302+0000 7f37eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:34.305 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.302+0000 7f37eeffd640 1 -- 172.21.15.66:0/2279971917 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f37e0001670 con 0x7f37f00b4d60 2024-03-25T18:11:34.305 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.302+0000 7f37eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:34.305 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.302+0000 7f37eeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:34.305 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.302+0000 7f37eeffd640 10 monclient: dump: 2024-03-25T18:11:34.305 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:34.305 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:34.305 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:34.305 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:34.305 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:34.305 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:34.305 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:34.305 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:34.305 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:34.306 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:34.306 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.302+0000 7f37f664a640 5 monclient: authenticate success, global_id 4561 2024-03-25T18:11:34.306 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.302+0000 7f37f664a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:34.315 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.314+0000 7f37eeffd640 1 -- 172.21.15.66:0/2279971917 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37e00017d0 con 0x7f37f00b4d60 2024-03-25T18:11:34.316 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.314+0000 7f37eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:34.316 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.314+0000 7f37eeffd640 1 -- 172.21.15.66:0/2279971917 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f37e0001be0 con 0x7f37f00b4d60 2024-03-25T18:11:34.316 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.314+0000 7f37eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:34.316 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.314+0000 7f37eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:34.316 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.314+0000 7f37eeffd640 10 monclient: dump: 2024-03-25T18:11:34.316 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:34.316 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:34.316 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:34.316 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:34.316 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:34.316 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:34.316 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:34.316 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:34.316 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:34.316 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:34.317 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.314+0000 7f37f664a640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:34.317 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.314+0000 7f37f664a640 4 set_mon_vals no callback set 2024-03-25T18:11:34.318 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.315+0000 7f37f664a640 10 monclient: shutdown 2024-03-25T18:11:34.318 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.315+0000 7f37f664a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:34.318 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.315+0000 7f37f664a640 1 -- 172.21.15.66:0/2279971917 >> v1:172.21.15.193:6789/0 conn(0x7f37f00b4d60 legacy=0x7f37f013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:34.318 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.315+0000 7f37f664a640 1 -- 172.21.15.66:0/2279971917 shutdown_connections 2024-03-25T18:11:34.318 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.315+0000 7f37f664a640 1 -- 172.21.15.66:0/2279971917 >> 172.21.15.66:0/2279971917 conn(0x7f37f00617a0 msgr2=0x7f37f0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:34.318 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.315+0000 7f37f664a640 1 -- 172.21.15.66:0/2279971917 shutdown_connections 2024-03-25T18:11:34.318 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.315+0000 7f37f664a640 1 -- 172.21.15.66:0/2279971917 wait complete. 2024-03-25T18:11:34.318 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.316+0000 7f37f664a640 10 monclient: build_initial_monmap 2024-03-25T18:11:34.319 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.316+0000 7f37f664a640 10 monclient: monmap: 2024-03-25T18:11:34.319 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:11:34.319 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:34.319 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:11:34.319 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:11:34.319 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:34.319 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:11:34.319 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:11:34.319 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:11:34.319 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:11:34.319 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:34.319 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.316+0000 7f37f664a640 1 librados: starting msgr at 2024-03-25T18:11:34.319 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.316+0000 7f37f664a640 1 librados: starting objecter 2024-03-25T18:11:34.319 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.316+0000 7f37f664a640 1 Processor -- start 2024-03-25T18:11:34.319 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.316+0000 7f37f664a640 1 -- start start 2024-03-25T18:11:34.319 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.316+0000 7f37f664a640 1 librados: setting wanted keys 2024-03-25T18:11:34.320 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.316+0000 7f37f664a640 1 librados: calling monclient init 2024-03-25T18:11:34.320 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.316+0000 7f37f664a640 10 monclient: init 2024-03-25T18:11:34.320 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.317+0000 7f37f664a640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:34.320 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.317+0000 7f37f664a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:11:34.321 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.317+0000 7f37f664a640 10 monclient(hunting): picked mon.noname-c con 0x7f37f00b1730 addr v1:172.21.15.66:6790/0 2024-03-25T18:11:34.321 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.317+0000 7f37f664a640 10 monclient(hunting): picked mon.noname-b con 0x7f37f00b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T18:11:34.321 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.317+0000 7f37f664a640 10 monclient(hunting): picked mon.noname-a con 0x7f37f014be90 addr v1:172.21.15.66:6789/0 2024-03-25T18:11:34.321 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.317+0000 7f37f664a640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f0138ed0 con 0x7f37f014be90 2024-03-25T18:11:34.321 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.317+0000 7f37f664a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f01413c0 con 0x7f37f00b4d60 2024-03-25T18:11:34.321 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.317+0000 7f37f664a640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f0143290 con 0x7f37f00b1730 2024-03-25T18:11:34.321 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.317+0000 7f37f664a640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:34.321 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.317+0000 7f37f664a640 10 monclient(hunting): authenticate will time out at 7443.377691s 2024-03-25T18:11:34.328 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.326+0000 7f37effff640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f37f00b1730 0x7f37f013c0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:42570/0 (socket says 172.21.15.66:42570) 2024-03-25T18:11:34.328 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.326+0000 7f37effff640 1 -- 172.21.15.66:0/1398388313 learned_addr learned my addr 172.21.15.66:0/1398388313 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:11:34.329 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.328+0000 7f37ecff9640 10 client.?.objecter ms_handle_connect 0x7f37f00b1730 2024-03-25T18:11:34.329 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.328+0000 7f37ecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:34.331 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.330+0000 7f37ecff9640 10 client.?.objecter ms_handle_connect 0x7f37f00b4d60 2024-03-25T18:11:34.332 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.330+0000 7f37ecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:34.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.337+0000 7f37ecff9640 10 client.?.objecter ms_handle_connect 0x7f37f014be90 2024-03-25T18:11:34.339 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.337+0000 7f37ecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:34.340 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.339+0000 7f37ecff9640 1 -- 172.21.15.66:0/1398388313 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 355955078 0 0) 0x7f37f0143290 con 0x7f37f00b1730 2024-03-25T18:11:34.340 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.339+0000 7f37ecff9640 10 monclient(hunting): _negotiate 2024-03-25T18:11:34.341 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.339+0000 7f37ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:34.341 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.339+0000 7f37ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:34.341 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.339+0000 7f37ecff9640 10 monclient(hunting): my global_id is 4565 2024-03-25T18:11:34.341 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.339+0000 7f37ecff9640 1 -- 172.21.15.66:0/1398388313 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37dc004eb0 con 0x7f37f00b1730 2024-03-25T18:11:34.341 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.339+0000 7f37ecff9640 1 -- 172.21.15.66:0/1398388313 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473632394 0 0) 0x7f37f01413c0 con 0x7f37f00b4d60 2024-03-25T18:11:34.341 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.339+0000 7f37ecff9640 10 monclient(hunting): _negotiate 2024-03-25T18:11:34.341 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.339+0000 7f37ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:34.341 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.339+0000 7f37ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:34.341 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.339+0000 7f37ecff9640 10 monclient(hunting): my global_id is 4564 2024-03-25T18:11:34.341 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.339+0000 7f37ecff9640 1 -- 172.21.15.66:0/1398388313 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37f0143290 con 0x7f37f00b4d60 2024-03-25T18:11:34.348 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.346+0000 7f37ecff9640 1 -- 172.21.15.66:0/1398388313 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895732498 0 0) 0x7f37f0138ed0 con 0x7f37f014be90 2024-03-25T18:11:34.348 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.346+0000 7f37ecff9640 10 monclient(hunting): _negotiate 2024-03-25T18:11:34.348 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.346+0000 7f37ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:34.348 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.346+0000 7f37ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:34.348 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.346+0000 7f37ecff9640 10 monclient(hunting): my global_id is 4569 2024-03-25T18:11:34.348 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.346+0000 7f37ecff9640 1 -- 172.21.15.66:0/1398388313 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37f01413c0 con 0x7f37f014be90 2024-03-25T18:11:34.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.350+0000 7f37ecff9640 1 -- 172.21.15.66:0/1398388313 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4215993058 0 0) 0x7f37dc004eb0 con 0x7f37f00b1730 2024-03-25T18:11:34.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.350+0000 7f37ecff9640 1 -- 172.21.15.66:0/1398388313 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37f0138ed0 con 0x7f37f00b1730 2024-03-25T18:11:34.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.350+0000 7f37ecff9640 1 -- 172.21.15.66:0/1398388313 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f37e0001670 con 0x7f37f00b1730 2024-03-25T18:11:34.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.350+0000 7f37ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:34.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.350+0000 7f37ecff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:34.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.350+0000 7f37ecff9640 10 monclient(hunting): dump: 2024-03-25T18:11:34.352 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:34.352 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:34.352 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:34.353 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:34.353 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:34.353 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:34.353 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:34.353 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:34.353 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:34.353 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:34.353 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.350+0000 7f37ecff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:34.362 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.360+0000 7f37ecff9640 1 -- 172.21.15.66:0/1398388313 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3528215295 0 0) 0x7f37f0143290 con 0x7f37f00b4d60 2024-03-25T18:11:34.362 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.360+0000 7f37ecff9640 1 -- 172.21.15.66:0/1398388313 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37dc004eb0 con 0x7f37f00b4d60 2024-03-25T18:11:34.362 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.360+0000 7f37ecff9640 1 -- 172.21.15.66:0/1398388313 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f37e4001870 con 0x7f37f00b4d60 2024-03-25T18:11:34.362 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.360+0000 7f37ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:34.362 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.360+0000 7f37ecff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:34.362 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.360+0000 7f37ecff9640 10 monclient(hunting): dump: 2024-03-25T18:11:34.362 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:34.362 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:34.362 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:34.362 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:34.362 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:34.362 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:34.362 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:34.362 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:34.363 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:34.363 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:34.384 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.381+0000 7f37ecff9640 1 -- 172.21.15.66:0/1398388313 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1547277125 0 0) 0x7f37f01413c0 con 0x7f37f014be90 2024-03-25T18:11:34.384 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.381+0000 7f37ecff9640 1 -- 172.21.15.66:0/1398388313 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37f0143290 con 0x7f37f014be90 2024-03-25T18:11:34.384 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.381+0000 7f37ecff9640 1 -- 172.21.15.66:0/1398388313 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f37e8001a30 con 0x7f37f014be90 2024-03-25T18:11:34.384 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.381+0000 7f37ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:34.384 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.381+0000 7f37ecff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:34.384 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.381+0000 7f37ecff9640 10 monclient(hunting): dump: 2024-03-25T18:11:34.384 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:34.384 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:34.384 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:34.384 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:34.384 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:34.385 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:34.385 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:34.385 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:34.385 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:34.385 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:34.385 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.381+0000 7f37ecff9640 1 -- 172.21.15.66:0/1398388313 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 840289695 0 0) 0x7f37dc004eb0 con 0x7f37f00b4d60 2024-03-25T18:11:34.385 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.381+0000 7f37ecff9640 1 -- 172.21.15.66:0/1398388313 >> v1:172.21.15.66:6790/0 conn(0x7f37f00b1730 legacy=0x7f37f013c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:34.385 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.382+0000 7f37ecff9640 1 -- 172.21.15.66:0/1398388313 >> v1:172.21.15.66:6789/0 conn(0x7f37f014be90 legacy=0x7f37f01e27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:34.385 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.382+0000 7f37ecff9640 10 monclient: _finish_hunting 0 2024-03-25T18:11:34.385 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.382+0000 7f37ecff9640 1 monclient: found mon.b 2024-03-25T18:11:34.385 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.382+0000 7f37ecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:34.385 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.382+0000 7f37ecff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:11:34.385 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.382+0000 7f37ecff9640 1 -- 172.21.15.66:0/1398388313 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f013e920 con 0x7f37f00b4d60 2024-03-25T18:11:34.385 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.382+0000 7f37ecff9640 10 monclient: _finish_auth 0 2024-03-25T18:11:34.385 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.382+0000 7f37ecff9640 10 monclient: _check_auth_tickets 2024-03-25T18:11:34.385 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.382+0000 7f37ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:34.386 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.382+0000 7f37f664a640 5 monclient: authenticate success, global_id 4564 2024-03-25T18:11:34.386 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.382+0000 7f37f664a640 10 monclient: _renew_subs 2024-03-25T18:11:34.386 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.382+0000 7f37f664a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:11:34.386 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.382+0000 7f37f664a640 1 -- 172.21.15.66:0/1398388313 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37f0140ea0 con 0x7f37f00b4d60 2024-03-25T18:11:34.386 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.382+0000 7f37f664a640 10 client.4564.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:34.386 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.382+0000 7f37f664a640 10 monclient: _renew_subs 2024-03-25T18:11:34.386 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.382+0000 7f37f664a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:11:34.386 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.382+0000 7f37f664a640 1 -- 172.21.15.66:0/1398388313 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f37f01422f0 con 0x7f37f00b4d60 2024-03-25T18:11:34.386 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.382+0000 7f37f664a640 1 librados: init done 2024-03-25T18:11:34.387 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.383+0000 7f37f664a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:34.387 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.383+0000 7f37f664a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:34.387 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.383+0000 7f37f664a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:11:34.387 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.383+0000 7f37f664a640 1 -- 172.21.15.66:0/1398388313 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37f0141570 con 0x7f37f00b4d60 2024-03-25T18:11:34.663 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.661+0000 7f37ecff9640 1 -- 172.21.15.66:0/1398388313 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37e40035f0 con 0x7f37f00b4d60 2024-03-25T18:11:34.663 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.661+0000 7f37ecff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:34.663 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.661+0000 7f37ecff9640 1 -- 172.21.15.66:0/1398388313 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f37e4001d80 con 0x7f37f00b4d60 2024-03-25T18:11:34.663 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.661+0000 7f37ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:34.663 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.661+0000 7f37ecff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:34.663 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.661+0000 7f37ecff9640 10 monclient: dump: 2024-03-25T18:11:34.663 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:34.663 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:34.664 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:34.664 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:34.664 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:34.664 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:34.664 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:34.664 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:34.664 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:34.664 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:34.664 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.661+0000 7f37ee7fc640 4 set_mon_vals no callback set 2024-03-25T18:11:34.665 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.662+0000 7f37ecff9640 1 -- 172.21.15.66:0/1398388313 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f37e40195d0 con 0x7f37f00b4d60 2024-03-25T18:11:34.665 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.663+0000 7f37effff640 10 monclient: get_auth_request con 0x7f37dc06e130 auth_method 0 2024-03-25T18:11:34.665 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.663+0000 7f37ecff9640 10 client.4564.objecter ms_handle_connect 0x7f37dc06e130 2024-03-25T18:11:34.670 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.669+0000 7f37ecff9640 1 -- 172.21.15.66:0/1398388313 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f37e40532b0 con 0x7f37f00b4d60 2024-03-25T18:11:34.670 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.669+0000 7f37ecff9640 10 client.4564.objecter ms_dispatch 0x7f37f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:34.670 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.669+0000 7f37ecff9640 3 client.4564.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:34.670 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.669+0000 7f37ecff9640 3 client.4564.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:34.670 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.669+0000 7f37ecff9640 20 client.4564.objecter dump_active .. 0 homeless 2024-03-25T18:11:34.671 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.669+0000 7f37ecff9640 1 -- 172.21.15.66:0/1398388313 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f37e404ebf0 con 0x7f37f00b4d60 2024-03-25T18:11:34.671 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.669+0000 7f37ecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:34.671 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.669+0000 7f37ecff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:34.890 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.888+0000 7f37f664a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:34.890 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.888+0000 7f37f664a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:34.890 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.888+0000 7f37f664a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:11:34.890 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.888+0000 7f37f664a640 1 -- 172.21.15.66:0/1398388313 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37f00627b0 con 0x7f37f00b4d60 2024-03-25T18:11:34.958 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.957+0000 7f37ecff9640 1 -- 172.21.15.66:0/1398388313 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f37e4050cf0 con 0x7f37f00b4d60 2024-03-25T18:11:34.959 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.957+0000 7f37ecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:34.959 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.957+0000 7f37ecff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:34.959 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:11:34.959 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:34.961 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.959+0000 7f37f664a640 10 librados: watch_flush enter 2024-03-25T18:11:34.962 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.959+0000 7f37f664a640 10 librados: watch_flush exit 2024-03-25T18:11:34.962 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.959+0000 7f37f664a640 20 client.4564.objecter shutdown clearing up homeless session... 2024-03-25T18:11:34.962 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.959+0000 7f37f664a640 10 client.4564.objecter successfully canceled tick 2024-03-25T18:11:34.962 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.959+0000 7f37f664a640 1 -- 172.21.15.66:0/1398388313 >> v1:172.21.15.193:6800/2275676803 conn(0x7f37dc06e130 legacy=0x7f37dc0705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:34.962 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.959+0000 7f37f664a640 10 monclient: shutdown 2024-03-25T18:11:34.962 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.959+0000 7f37f664a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:34.962 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.959+0000 7f37f664a640 1 -- 172.21.15.66:0/1398388313 >> v1:172.21.15.193:6789/0 conn(0x7f37f00b4d60 legacy=0x7f37f0141a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:34.962 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.960+0000 7f37f664a640 1 -- 172.21.15.66:0/1398388313 shutdown_connections 2024-03-25T18:11:34.962 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.960+0000 7f37f664a640 1 -- 172.21.15.66:0/1398388313 >> 172.21.15.66:0/1398388313 conn(0x7f37f00617a0 msgr2=0x7f37f0143900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:34.962 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.960+0000 7f37f664a640 1 -- 172.21.15.66:0/1398388313 shutdown_connections 2024-03-25T18:11:34.962 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.960+0000 7f37f664a640 1 -- 172.21.15.66:0/1398388313 wait complete. 2024-03-25T18:11:34.962 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:34.960+0000 7f37f664a640 1 librados: shutdown 2024-03-25T18:11:34.972 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:41.779 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:41.900 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.900+0000 7fa423b3d640 10 monclient: get_monmap_and_config 2024-03-25T18:11:41.900 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.900+0000 7fa423b3d640 10 monclient: build_initial_monmap 2024-03-25T18:11:41.901 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.900+0000 7fa423b3d640 10 monclient: monmap: 2024-03-25T18:11:41.901 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:11:41.901 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:41.901 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:11:41.900817+0000 2024-03-25T18:11:41.901 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:11:41.900817+0000 2024-03-25T18:11:41.901 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:41.901 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:41.901 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:11:41.901 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:11:41.901 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:11:41.901 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:41.902 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.901+0000 7fa423b3d640 1 Processor -- start 2024-03-25T18:11:41.903 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.901+0000 7fa423b3d640 1 -- start start 2024-03-25T18:11:41.903 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.901+0000 7fa423b3d640 10 monclient: init 2024-03-25T18:11:41.903 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.902+0000 7fa423b3d640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:41.903 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.902+0000 7fa423b3d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:11:41.903 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.902+0000 7fa423b3d640 10 monclient(hunting): picked mon.noname-a con 0x7fa41c0b4d60 addr v1:172.21.15.66:6789/0 2024-03-25T18:11:41.903 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.902+0000 7fa423b3d640 10 monclient(hunting): picked mon.noname-c con 0x7fa41c0b1730 addr v1:172.21.15.193:6789/0 2024-03-25T18:11:41.903 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.902+0000 7fa423b3d640 10 monclient(hunting): picked mon.noname-b con 0x7fa41c14be90 addr v1:172.21.15.66:6790/0 2024-03-25T18:11:41.903 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.902+0000 7fa423b3d640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa41c0af940 con 0x7fa41c0b4d60 2024-03-25T18:11:41.903 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.902+0000 7fa423b3d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa41c0b0a70 con 0x7fa41c0b1730 2024-03-25T18:11:41.903 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.902+0000 7fa423b3d640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa41c0b0d50 con 0x7fa41c14be90 2024-03-25T18:11:41.903 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.902+0000 7fa423b3d640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:41.903 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.902+0000 7fa423b3d640 10 monclient(hunting): authenticate will time out at 7450.961312s 2024-03-25T18:11:41.911 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.911+0000 7fa4220da640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7fa41c14be90 0x7fa41c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:47038/0 (socket says 172.21.15.66:47038) 2024-03-25T18:11:41.912 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.911+0000 7fa4220da640 1 -- 172.21.15.66:0/664000027 learned_addr learned my addr 172.21.15.66:0/664000027 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:11:41.923 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.923+0000 7fa4208d7640 1 -- 172.21.15.66:0/664000027 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1699368663 0 0) 0x7fa41c0b0d50 con 0x7fa41c14be90 2024-03-25T18:11:41.923 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.923+0000 7fa4208d7640 10 monclient(hunting): _negotiate 2024-03-25T18:11:41.923 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.923+0000 7fa4208d7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:41.923 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.923+0000 7fa4208d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:41.924 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.923+0000 7fa4208d7640 10 monclient(hunting): my global_id is 4568 2024-03-25T18:11:41.924 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.923+0000 7fa4208d7640 1 -- 172.21.15.66:0/664000027 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa408003520 con 0x7fa41c14be90 2024-03-25T18:11:41.928 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.927+0000 7fa4208d7640 1 -- 172.21.15.66:0/664000027 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1409504123 0 0) 0x7fa408003520 con 0x7fa41c14be90 2024-03-25T18:11:41.928 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.927+0000 7fa4208d7640 1 -- 172.21.15.66:0/664000027 >> v1:172.21.15.193:6789/0 conn(0x7fa41c0b1730 legacy=0x7fa41c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:41.928 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.927+0000 7fa4208d7640 1 -- 172.21.15.66:0/664000027 >> v1:172.21.15.66:6789/0 conn(0x7fa41c0b4d60 legacy=0x7fa41c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:41.928 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.927+0000 7fa4208d7640 10 monclient: _finish_hunting 0 2024-03-25T18:11:41.928 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.927+0000 7fa4208d7640 1 monclient: found mon.noname-b 2024-03-25T18:11:41.928 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.927+0000 7fa4208d7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:41.928 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.927+0000 7fa4208d7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.66:6790/0 2024-03-25T18:11:41.928 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.927+0000 7fa4208d7640 1 -- 172.21.15.66:0/664000027 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa41c0b1030 con 0x7fa41c14be90 2024-03-25T18:11:41.928 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.927+0000 7fa4208d7640 10 monclient: _finish_auth 0 2024-03-25T18:11:41.928 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.927+0000 7fa4208d7640 10 monclient: _check_auth_tickets 2024-03-25T18:11:41.929 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.927+0000 7fa4208d7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:41.929 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.927+0000 7fa4208d7640 1 -- 172.21.15.66:0/664000027 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fa414001a10 con 0x7fa41c14be90 2024-03-25T18:11:41.929 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.927+0000 7fa423b3d640 5 monclient: authenticate success, global_id 4568 2024-03-25T18:11:41.929 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.927+0000 7fa423b3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:41.929 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.927+0000 7fa4208d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:41.929 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.927+0000 7fa4208d7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:41.929 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.927+0000 7fa4208d7640 10 monclient: dump: 2024-03-25T18:11:41.929 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:41.929 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:41.929 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:41.929 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:41.929 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:41.929 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:41.929 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:41.929 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:41.930 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:41.930 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:41.930 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.927+0000 7fa423b3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:41.939 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.938+0000 7fa4208d7640 1 -- 172.21.15.66:0/664000027 <== mon.2 v1:172.21.15.66:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa414001b70 con 0x7fa41c14be90 2024-03-25T18:11:41.939 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.938+0000 7fa4208d7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:41.939 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.938+0000 7fa4208d7640 1 -- 172.21.15.66:0/664000027 <== mon.2 v1:172.21.15.66:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fa414001f80 con 0x7fa41c14be90 2024-03-25T18:11:41.939 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.938+0000 7fa423b3d640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:41.939 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.938+0000 7fa423b3d640 4 set_mon_vals no callback set 2024-03-25T18:11:41.939 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.938+0000 7fa4208d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:41.939 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.938+0000 7fa4208d7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:11:41.939 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.938+0000 7fa4208d7640 10 monclient: dump: 2024-03-25T18:11:41.939 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:41.939 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:41.939 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:41.940 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:41.940 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:41.940 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:41.940 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:41.940 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:41.940 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:41.940 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:41.941 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.939+0000 7fa423b3d640 10 monclient: shutdown 2024-03-25T18:11:41.941 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.939+0000 7fa423b3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:41.941 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.939+0000 7fa423b3d640 1 -- 172.21.15.66:0/664000027 >> v1:172.21.15.66:6790/0 conn(0x7fa41c14be90 legacy=0x7fa41c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:41.942 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.939+0000 7fa423b3d640 1 -- 172.21.15.66:0/664000027 shutdown_connections 2024-03-25T18:11:41.942 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.939+0000 7fa423b3d640 1 -- 172.21.15.66:0/664000027 >> 172.21.15.66:0/664000027 conn(0x7fa41c0617a0 msgr2=0x7fa41c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:41.942 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.939+0000 7fa423b3d640 1 -- 172.21.15.66:0/664000027 shutdown_connections 2024-03-25T18:11:41.942 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.939+0000 7fa423b3d640 1 -- 172.21.15.66:0/664000027 wait complete. 2024-03-25T18:11:41.942 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.940+0000 7fa423b3d640 10 monclient: build_initial_monmap 2024-03-25T18:11:41.942 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.940+0000 7fa423b3d640 10 monclient: monmap: 2024-03-25T18:11:41.942 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:11:41.942 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:41.942 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:11:41.942 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:11:41.942 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:41.942 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:11:41.945 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:11:41.945 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:11:41.946 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:11:41.946 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:41.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.940+0000 7fa423b3d640 1 librados: starting msgr at 2024-03-25T18:11:41.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.940+0000 7fa423b3d640 1 librados: starting objecter 2024-03-25T18:11:41.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.941+0000 7fa423b3d640 1 Processor -- start 2024-03-25T18:11:41.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.941+0000 7fa423b3d640 1 -- start start 2024-03-25T18:11:41.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.941+0000 7fa423b3d640 1 librados: setting wanted keys 2024-03-25T18:11:41.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.941+0000 7fa423b3d640 1 librados: calling monclient init 2024-03-25T18:11:41.946 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.941+0000 7fa423b3d640 10 monclient: init 2024-03-25T18:11:41.947 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.941+0000 7fa423b3d640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:41.947 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.941+0000 7fa423b3d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:11:41.947 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.941+0000 7fa423b3d640 10 monclient(hunting): picked mon.noname-c con 0x7fa41c0b1730 addr v1:172.21.15.66:6790/0 2024-03-25T18:11:41.947 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.941+0000 7fa423b3d640 10 monclient(hunting): picked mon.noname-a con 0x7fa41c0b4d60 addr v1:172.21.15.66:6789/0 2024-03-25T18:11:41.947 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.942+0000 7fa423b3d640 10 monclient(hunting): picked mon.noname-b con 0x7fa41c14be90 addr v1:172.21.15.193:6789/0 2024-03-25T18:11:41.947 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.942+0000 7fa423b3d640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa41c141d30 con 0x7fa41c0b4d60 2024-03-25T18:11:41.947 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.942+0000 7fa423b3d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa41c140960 con 0x7fa41c14be90 2024-03-25T18:11:41.947 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.942+0000 7fa423b3d640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa41c137fd0 con 0x7fa41c0b1730 2024-03-25T18:11:41.948 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.942+0000 7fa423b3d640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:41.948 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.942+0000 7fa423b3d640 10 monclient(hunting): authenticate will time out at 7451.001173s 2024-03-25T18:11:41.953 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.953+0000 7fa4220da640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fa41c14be90 0x7fa41c1e2ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:45188/0 (socket says 172.21.15.66:45188) 2024-03-25T18:11:41.953 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.953+0000 7fa4220da640 1 -- 172.21.15.66:0/3303684264 learned_addr learned my addr 172.21.15.66:0/3303684264 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:11:41.954 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.954+0000 7fa4127fc640 10 client.?.objecter ms_handle_connect 0x7fa41c14be90 2024-03-25T18:11:41.954 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.954+0000 7fa4127fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:41.956 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.955+0000 7fa4127fc640 10 client.?.objecter ms_handle_connect 0x7fa41c0b1730 2024-03-25T18:11:41.956 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.955+0000 7fa4127fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:41.956 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.956+0000 7fa4127fc640 10 client.?.objecter ms_handle_connect 0x7fa41c0b4d60 2024-03-25T18:11:41.956 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.956+0000 7fa4127fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:41.961 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.961+0000 7fa4127fc640 1 -- 172.21.15.66:0/3303684264 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2103539645 0 0) 0x7fa41c140960 con 0x7fa41c14be90 2024-03-25T18:11:41.961 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.961+0000 7fa4127fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:41.961 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.961+0000 7fa4127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:41.961 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.961+0000 7fa4127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:41.962 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.961+0000 7fa4127fc640 10 monclient(hunting): my global_id is 4570 2024-03-25T18:11:41.962 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.961+0000 7fa4127fc640 1 -- 172.21.15.66:0/3303684264 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa400004eb0 con 0x7fa41c14be90 2024-03-25T18:11:41.974 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.974+0000 7fa4127fc640 1 -- 172.21.15.66:0/3303684264 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 825196971 0 0) 0x7fa400004eb0 con 0x7fa41c14be90 2024-03-25T18:11:41.974 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.974+0000 7fa4127fc640 1 -- 172.21.15.66:0/3303684264 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa41c140960 con 0x7fa41c14be90 2024-03-25T18:11:41.974 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.974+0000 7fa4127fc640 1 -- 172.21.15.66:0/3303684264 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fa414001f40 con 0x7fa41c14be90 2024-03-25T18:11:41.974 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.974+0000 7fa4127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:41.974 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.974+0000 7fa4127fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:41.974 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.974+0000 7fa4127fc640 10 monclient(hunting): dump: 2024-03-25T18:11:41.975 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:41.975 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:41.975 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:41.975 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:41.975 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:41.975 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:41.975 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:41.975 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:41.975 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:41.975 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:41.975 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.974+0000 7fa4127fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:41.978 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.977+0000 7fa4127fc640 1 -- 172.21.15.66:0/3303684264 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1259684081 0 0) 0x7fa41c141d30 con 0x7fa41c0b4d60 2024-03-25T18:11:41.978 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.977+0000 7fa4127fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:41.978 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.977+0000 7fa4127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:41.978 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.977+0000 7fa4127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:41.978 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.977+0000 7fa4127fc640 10 monclient(hunting): my global_id is 4575 2024-03-25T18:11:41.978 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.977+0000 7fa4127fc640 1 -- 172.21.15.66:0/3303684264 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa400004eb0 con 0x7fa41c0b4d60 2024-03-25T18:11:41.978 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.977+0000 7fa4127fc640 1 -- 172.21.15.66:0/3303684264 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3166139527 0 0) 0x7fa41c137fd0 con 0x7fa41c0b1730 2024-03-25T18:11:41.978 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.977+0000 7fa4127fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:41.978 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.977+0000 7fa4127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:41.979 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.977+0000 7fa4127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:41.979 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.977+0000 7fa4127fc640 10 monclient(hunting): my global_id is 4571 2024-03-25T18:11:41.979 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.977+0000 7fa4127fc640 1 -- 172.21.15.66:0/3303684264 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa41c141d30 con 0x7fa41c0b1730 2024-03-25T18:11:41.988 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.987+0000 7fa4127fc640 1 -- 172.21.15.66:0/3303684264 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2061042501 0 0) 0x7fa400004eb0 con 0x7fa41c0b4d60 2024-03-25T18:11:41.988 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.987+0000 7fa4127fc640 1 -- 172.21.15.66:0/3303684264 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa41c137fd0 con 0x7fa41c0b4d60 2024-03-25T18:11:41.988 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.987+0000 7fa4127fc640 1 -- 172.21.15.66:0/3303684264 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fa40c001830 con 0x7fa41c0b4d60 2024-03-25T18:11:41.988 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.987+0000 7fa4127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:41.988 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.987+0000 7fa4127fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:41.988 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.987+0000 7fa4127fc640 10 monclient(hunting): dump: 2024-03-25T18:11:41.988 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:41.988 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:41.988 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:41.988 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:41.988 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:41.988 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:41.988 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:41.988 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:41.989 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:41.989 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:41.994 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.993+0000 7fa4127fc640 1 -- 172.21.15.66:0/3303684264 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1989037557 0 0) 0x7fa41c141d30 con 0x7fa41c0b1730 2024-03-25T18:11:41.994 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.993+0000 7fa4127fc640 1 -- 172.21.15.66:0/3303684264 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa400004eb0 con 0x7fa41c0b1730 2024-03-25T18:11:41.994 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.993+0000 7fa4127fc640 1 -- 172.21.15.66:0/3303684264 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fa418001860 con 0x7fa41c0b1730 2024-03-25T18:11:41.994 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.993+0000 7fa4127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:41.994 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.993+0000 7fa4127fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:11:41.994 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.993+0000 7fa4127fc640 10 monclient(hunting): dump: 2024-03-25T18:11:41.994 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:41.994 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:41.994 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:41.994 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:41.995 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:41.995 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:41.995 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:41.995 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:41.995 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:41.995 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:41.996 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.994+0000 7fa4127fc640 1 -- 172.21.15.66:0/3303684264 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3928530490 0 0) 0x7fa41c140960 con 0x7fa41c14be90 2024-03-25T18:11:41.996 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.994+0000 7fa4127fc640 1 -- 172.21.15.66:0/3303684264 >> v1:172.21.15.66:6790/0 conn(0x7fa41c0b1730 legacy=0x7fa41c131cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:41.996 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.994+0000 7fa4127fc640 1 -- 172.21.15.66:0/3303684264 >> v1:172.21.15.66:6789/0 conn(0x7fa41c0b4d60 legacy=0x7fa41c142430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:41.996 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.994+0000 7fa4127fc640 10 monclient: _finish_hunting 0 2024-03-25T18:11:41.996 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.994+0000 7fa4127fc640 1 monclient: found mon.b 2024-03-25T18:11:41.997 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.994+0000 7fa4127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:41.997 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.994+0000 7fa4127fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:11:41.997 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.994+0000 7fa4127fc640 1 -- 172.21.15.66:0/3303684264 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa41c138160 con 0x7fa41c14be90 2024-03-25T18:11:41.997 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.994+0000 7fa4127fc640 10 monclient: _finish_auth 0 2024-03-25T18:11:41.997 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.994+0000 7fa4127fc640 10 monclient: _check_auth_tickets 2024-03-25T18:11:41.997 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.994+0000 7fa4127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:41.997 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.994+0000 7fa423b3d640 5 monclient: authenticate success, global_id 4570 2024-03-25T18:11:41.997 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.994+0000 7fa423b3d640 10 monclient: _renew_subs 2024-03-25T18:11:41.997 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.994+0000 7fa423b3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:11:41.997 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.994+0000 7fa423b3d640 1 -- 172.21.15.66:0/3303684264 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa41c140380 con 0x7fa41c14be90 2024-03-25T18:11:41.997 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.994+0000 7fa423b3d640 10 client.4570.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:41.997 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.994+0000 7fa423b3d640 10 monclient: _renew_subs 2024-03-25T18:11:41.997 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.994+0000 7fa423b3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:11:41.997 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.994+0000 7fa423b3d640 1 -- 172.21.15.66:0/3303684264 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa41c1453b0 con 0x7fa41c14be90 2024-03-25T18:11:41.997 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.994+0000 7fa423b3d640 1 librados: init done 2024-03-25T18:11:41.997 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.995+0000 7fa423b3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:41.998 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.995+0000 7fa423b3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:41.998 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.995+0000 7fa423b3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:11:41.998 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:41.995+0000 7fa423b3d640 1 -- 172.21.15.66:0/3303684264 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa41c140f10 con 0x7fa41c14be90 2024-03-25T18:11:42.278 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.277+0000 7fa4127fc640 1 -- 172.21.15.66:0/3303684264 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa414004e10 con 0x7fa41c14be90 2024-03-25T18:11:42.278 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.277+0000 7fa4127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:42.278 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.277+0000 7fa4127fc640 1 -- 172.21.15.66:0/3303684264 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fa414004f80 con 0x7fa41c14be90 2024-03-25T18:11:42.278 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.277+0000 7fa4127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:42.278 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.277+0000 7fa4127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:42.278 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.277+0000 7fa4127fc640 10 monclient: dump: 2024-03-25T18:11:42.278 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:42.278 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:42.278 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:42.278 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:42.279 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:42.279 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:42.279 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:42.279 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:42.279 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:42.279 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:42.279 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.277+0000 7fa413fff640 4 set_mon_vals no callback set 2024-03-25T18:11:42.280 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.278+0000 7fa4127fc640 1 -- 172.21.15.66:0/3303684264 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7fa41401af50 con 0x7fa41c14be90 2024-03-25T18:11:42.280 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.279+0000 7fa4218d9640 10 monclient: get_auth_request con 0x7fa40006e180 auth_method 0 2024-03-25T18:11:42.280 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.279+0000 7fa4127fc640 10 client.4570.objecter ms_handle_connect 0x7fa40006e180 2024-03-25T18:11:42.283 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.283+0000 7fa4127fc640 1 -- 172.21.15.66:0/3303684264 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7fa414054990 con 0x7fa41c14be90 2024-03-25T18:11:42.283 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.283+0000 7fa4127fc640 10 client.4570.objecter ms_dispatch 0x7fa41c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:42.284 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.283+0000 7fa4127fc640 3 client.4570.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:42.284 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.283+0000 7fa4127fc640 3 client.4570.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:42.284 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.283+0000 7fa4127fc640 20 client.4570.objecter dump_active .. 0 homeless 2024-03-25T18:11:42.284 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.283+0000 7fa4127fc640 1 -- 172.21.15.66:0/3303684264 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa4140504a0 con 0x7fa41c14be90 2024-03-25T18:11:42.284 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.283+0000 7fa4127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:42.284 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.283+0000 7fa4127fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:42.546 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.545+0000 7fa423b3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:42.546 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.545+0000 7fa423b3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:42.546 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.545+0000 7fa423b3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:11:42.546 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.545+0000 7fa423b3d640 1 -- 172.21.15.66:0/3303684264 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa41c140af0 con 0x7fa41c14be90 2024-03-25T18:11:42.582 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.581+0000 7fa4127fc640 1 -- 172.21.15.66:0/3303684264 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7fa4140523d0 con 0x7fa41c14be90 2024-03-25T18:11:42.582 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.581+0000 7fa4127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:42.582 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.581+0000 7fa4127fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:42.582 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:11:42.582 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:42.585 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.583+0000 7fa423b3d640 10 librados: watch_flush enter 2024-03-25T18:11:42.585 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.583+0000 7fa423b3d640 10 librados: watch_flush exit 2024-03-25T18:11:42.585 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.583+0000 7fa423b3d640 20 client.4570.objecter shutdown clearing up homeless session... 2024-03-25T18:11:42.585 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.583+0000 7fa423b3d640 10 client.4570.objecter successfully canceled tick 2024-03-25T18:11:42.585 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.583+0000 7fa423b3d640 1 -- 172.21.15.66:0/3303684264 >> v1:172.21.15.193:6800/2275676803 conn(0x7fa40006e180 legacy=0x7fa400070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:42.585 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.583+0000 7fa423b3d640 10 monclient: shutdown 2024-03-25T18:11:42.585 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.583+0000 7fa423b3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:42.585 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.583+0000 7fa423b3d640 1 -- 172.21.15.66:0/3303684264 >> v1:172.21.15.193:6789/0 conn(0x7fa41c14be90 legacy=0x7fa41c1e2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:42.585 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.584+0000 7fa423b3d640 1 -- 172.21.15.66:0/3303684264 shutdown_connections 2024-03-25T18:11:42.585 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.584+0000 7fa423b3d640 1 -- 172.21.15.66:0/3303684264 >> 172.21.15.66:0/3303684264 conn(0x7fa41c0617a0 msgr2=0x7fa41c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:42.585 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.584+0000 7fa423b3d640 1 -- 172.21.15.66:0/3303684264 shutdown_connections 2024-03-25T18:11:42.585 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.584+0000 7fa423b3d640 1 -- 172.21.15.66:0/3303684264 wait complete. 2024-03-25T18:11:42.585 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:42.584+0000 7fa423b3d640 1 librados: shutdown 2024-03-25T18:11:42.596 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:49.403 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:49.526 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.525+0000 7f7ea85a3640 10 monclient: get_monmap_and_config 2024-03-25T18:11:49.526 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.525+0000 7f7ea85a3640 10 monclient: build_initial_monmap 2024-03-25T18:11:49.526 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.525+0000 7f7ea85a3640 10 monclient: monmap: 2024-03-25T18:11:49.526 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:11:49.526 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:49.526 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:11:49.526114+0000 2024-03-25T18:11:49.526 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:11:49.526114+0000 2024-03-25T18:11:49.526 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:49.526 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:49.527 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:11:49.527 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:11:49.527 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:11:49.527 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:49.528 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.526+0000 7f7ea85a3640 1 Processor -- start 2024-03-25T18:11:49.528 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.526+0000 7f7ea85a3640 1 -- start start 2024-03-25T18:11:49.528 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.526+0000 7f7ea85a3640 10 monclient: init 2024-03-25T18:11:49.528 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.527+0000 7f7ea85a3640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:49.528 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.527+0000 7f7ea85a3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:11:49.528 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.527+0000 7f7ea85a3640 10 monclient(hunting): picked mon.noname-b con 0x7f7ea00a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:11:49.528 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.527+0000 7f7ea85a3640 10 monclient(hunting): picked mon.noname-c con 0x7f7ea013f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:11:49.528 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.527+0000 7f7ea85a3640 10 monclient(hunting): picked mon.noname-a con 0x7f7ea01430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:11:49.528 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.527+0000 7f7ea85a3640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ea0142520 con 0x7f7ea01430c0 2024-03-25T18:11:49.529 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.527+0000 7f7ea85a3640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ea00b28a0 con 0x7f7ea013f710 2024-03-25T18:11:49.529 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.527+0000 7f7ea85a3640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ea00b2a30 con 0x7f7ea00a98f0 2024-03-25T18:11:49.529 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.527+0000 7f7ea85a3640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:49.529 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.527+0000 7f7ea85a3640 10 monclient(hunting): authenticate will time out at 7458.586538s 2024-03-25T18:11:49.535 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.534+0000 7f7ea5b3e640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f7ea013f710 0x7f7ea01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:36714/0 (socket says 172.21.15.66:36714) 2024-03-25T18:11:49.535 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.534+0000 7f7ea5b3e640 1 -- 172.21.15.66:0/2142265037 learned_addr learned my addr 172.21.15.66:0/2142265037 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:11:49.615 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.615+0000 7f7ea533d640 1 -- 172.21.15.66:0/2142265037 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 735171311 0 0) 0x7f7ea0142520 con 0x7f7ea01430c0 2024-03-25T18:11:49.615 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.615+0000 7f7ea533d640 10 monclient(hunting): _negotiate 2024-03-25T18:11:49.615 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.615+0000 7f7ea533d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:49.615 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.615+0000 7f7ea533d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:49.615 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.615+0000 7f7ea533d640 10 monclient(hunting): my global_id is 4578 2024-03-25T18:11:49.616 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.615+0000 7f7ea533d640 1 -- 172.21.15.66:0/2142265037 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e8c003520 con 0x7f7ea01430c0 2024-03-25T18:11:49.629 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.628+0000 7f7ea533d640 1 -- 172.21.15.66:0/2142265037 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2727263372 0 0) 0x7f7e8c003520 con 0x7f7ea01430c0 2024-03-25T18:11:49.629 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.628+0000 7f7ea533d640 1 -- 172.21.15.66:0/2142265037 >> v1:172.21.15.66:6790/0 conn(0x7f7ea00a98f0 legacy=0x7f7ea00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:49.629 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.628+0000 7f7ea533d640 1 -- 172.21.15.66:0/2142265037 >> v1:172.21.15.193:6789/0 conn(0x7f7ea013f710 legacy=0x7f7ea01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:49.629 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.628+0000 7f7ea533d640 10 monclient: _finish_hunting 0 2024-03-25T18:11:49.629 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.628+0000 7f7ea533d640 1 monclient: found mon.noname-a 2024-03-25T18:11:49.630 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.628+0000 7f7ea533d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:49.630 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.628+0000 7f7ea533d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.66:6789/0 2024-03-25T18:11:49.630 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.628+0000 7f7ea533d640 1 -- 172.21.15.66:0/2142265037 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea00b44c0 con 0x7f7ea01430c0 2024-03-25T18:11:49.630 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.628+0000 7f7ea533d640 10 monclient: _finish_auth 0 2024-03-25T18:11:49.630 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.628+0000 7f7ea533d640 10 monclient: _check_auth_tickets 2024-03-25T18:11:49.630 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.628+0000 7f7ea533d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:49.630 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.628+0000 7f7ea533d640 1 -- 172.21.15.66:0/2142265037 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7e98001a10 con 0x7f7ea01430c0 2024-03-25T18:11:49.630 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.628+0000 7f7ea85a3640 5 monclient: authenticate success, global_id 4578 2024-03-25T18:11:49.630 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.628+0000 7f7ea85a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:49.630 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.628+0000 7f7ea533d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:49.630 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.628+0000 7f7ea533d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:49.630 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.628+0000 7f7ea533d640 10 monclient: dump: 2024-03-25T18:11:49.630 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:49.630 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:49.631 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:49.631 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:49.631 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:49.631 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:49.631 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:49.631 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:49.631 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:49.631 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:49.631 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.628+0000 7f7ea85a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:49.651 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.650+0000 7f7ea533d640 1 -- 172.21.15.66:0/2142265037 <== mon.0 v1:172.21.15.66:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e98001b70 con 0x7f7ea01430c0 2024-03-25T18:11:49.651 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.650+0000 7f7ea533d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:49.651 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.650+0000 7f7ea533d640 1 -- 172.21.15.66:0/2142265037 <== mon.0 v1:172.21.15.66:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7e98001f80 con 0x7f7ea01430c0 2024-03-25T18:11:49.651 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.650+0000 7f7ea85a3640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:49.651 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.650+0000 7f7ea85a3640 4 set_mon_vals no callback set 2024-03-25T18:11:49.651 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.650+0000 7f7ea533d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:49.651 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.650+0000 7f7ea533d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:49.652 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.650+0000 7f7ea533d640 10 monclient: dump: 2024-03-25T18:11:49.652 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:49.652 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:49.652 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:49.652 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:49.652 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:49.652 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:49.652 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:49.652 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:49.652 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:49.652 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:49.654 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.651+0000 7f7ea85a3640 10 monclient: shutdown 2024-03-25T18:11:49.654 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.651+0000 7f7ea85a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:49.654 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.651+0000 7f7ea85a3640 1 -- 172.21.15.66:0/2142265037 >> v1:172.21.15.66:6789/0 conn(0x7f7ea01430c0 legacy=0x7f7ea014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:49.654 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.651+0000 7f7ea85a3640 1 -- 172.21.15.66:0/2142265037 shutdown_connections 2024-03-25T18:11:49.654 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.651+0000 7f7ea85a3640 1 -- 172.21.15.66:0/2142265037 >> 172.21.15.66:0/2142265037 conn(0x7f7ea00b7ea0 msgr2=0x7f7ea00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:49.654 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.652+0000 7f7ea85a3640 1 -- 172.21.15.66:0/2142265037 shutdown_connections 2024-03-25T18:11:49.654 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.652+0000 7f7ea85a3640 1 -- 172.21.15.66:0/2142265037 wait complete. 2024-03-25T18:11:49.654 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.652+0000 7f7ea85a3640 10 monclient: build_initial_monmap 2024-03-25T18:11:49.654 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.652+0000 7f7ea85a3640 10 monclient: monmap: 2024-03-25T18:11:49.654 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:11:49.654 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:49.654 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:11:49.654 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:11:49.654 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:49.655 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:11:49.655 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:11:49.655 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:11:49.655 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:11:49.655 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:49.655 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.653+0000 7f7ea85a3640 1 librados: starting msgr at 2024-03-25T18:11:49.655 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.653+0000 7f7ea85a3640 1 librados: starting objecter 2024-03-25T18:11:49.655 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.653+0000 7f7ea85a3640 1 Processor -- start 2024-03-25T18:11:49.655 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.653+0000 7f7ea85a3640 1 -- start start 2024-03-25T18:11:49.655 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.653+0000 7f7ea85a3640 1 librados: setting wanted keys 2024-03-25T18:11:49.655 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.653+0000 7f7ea85a3640 1 librados: calling monclient init 2024-03-25T18:11:49.655 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.653+0000 7f7ea85a3640 10 monclient: init 2024-03-25T18:11:49.656 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.654+0000 7f7ea85a3640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:49.656 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.654+0000 7f7ea85a3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:11:49.656 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.654+0000 7f7ea85a3640 10 monclient(hunting): picked mon.noname-a con 0x7f7ea00a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:11:49.656 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.654+0000 7f7ea85a3640 10 monclient(hunting): picked mon.noname-b con 0x7f7ea013f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:11:49.656 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.654+0000 7f7ea85a3640 10 monclient(hunting): picked mon.noname-c con 0x7f7ea01430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:11:49.656 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.654+0000 7f7ea85a3640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ea0137f10 con 0x7f7ea00a98f0 2024-03-25T18:11:49.657 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.654+0000 7f7ea85a3640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ea01dc070 con 0x7f7ea013f710 2024-03-25T18:11:49.657 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.654+0000 7f7ea85a3640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ea0137100 con 0x7f7ea01430c0 2024-03-25T18:11:49.657 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.654+0000 7f7ea85a3640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:49.657 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.654+0000 7f7ea85a3640 10 monclient(hunting): authenticate will time out at 7458.713718s 2024-03-25T18:11:49.665 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.665+0000 7f7ea5b3e640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f7ea013f710 0x7f7ea01db250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:36748/0 (socket says 172.21.15.66:36748) 2024-03-25T18:11:49.665 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.665+0000 7f7ea5b3e640 1 -- 172.21.15.66:0/4243229791 learned_addr learned my addr 172.21.15.66:0/4243229791 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:11:49.666 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.666+0000 7f7e96ffd640 10 client.?.objecter ms_handle_connect 0x7f7ea013f710 2024-03-25T18:11:49.667 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.666+0000 7f7e96ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:49.667 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.667+0000 7f7e96ffd640 10 client.?.objecter ms_handle_connect 0x7f7ea00a98f0 2024-03-25T18:11:49.667 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.667+0000 7f7e96ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:49.676 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.675+0000 7f7e96ffd640 1 -- 172.21.15.66:0/4243229791 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3672415375 0 0) 0x7f7ea01dc070 con 0x7f7ea013f710 2024-03-25T18:11:49.676 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.676+0000 7f7e96ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:49.676 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.676+0000 7f7e96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:49.676 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.676+0000 7f7e96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:49.676 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.676+0000 7f7e96ffd640 10 monclient(hunting): my global_id is 4576 2024-03-25T18:11:49.676 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.676+0000 7f7e96ffd640 1 -- 172.21.15.66:0/4243229791 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e84004eb0 con 0x7f7ea013f710 2024-03-25T18:11:49.682 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.682+0000 7f7e96ffd640 1 -- 172.21.15.66:0/4243229791 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3891886927 0 0) 0x7f7ea0137f10 con 0x7f7ea00a98f0 2024-03-25T18:11:49.682 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.682+0000 7f7e96ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:49.682 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.682+0000 7f7e96ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:49.682 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.682+0000 7f7e96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:49.682 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.682+0000 7f7e96ffd640 10 monclient(hunting): my global_id is 4581 2024-03-25T18:11:49.682 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.682+0000 7f7e96ffd640 1 -- 172.21.15.66:0/4243229791 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ea01dc070 con 0x7f7ea00a98f0 2024-03-25T18:11:49.683 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.683+0000 7f7e96ffd640 10 client.?.objecter ms_handle_connect 0x7f7ea01430c0 2024-03-25T18:11:49.683 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.683+0000 7f7e96ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:49.689 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.688+0000 7f7e96ffd640 1 -- 172.21.15.66:0/4243229791 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 228576156 0 0) 0x7f7e84004eb0 con 0x7f7ea013f710 2024-03-25T18:11:49.689 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.688+0000 7f7e96ffd640 1 -- 172.21.15.66:0/4243229791 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ea0137f10 con 0x7f7ea013f710 2024-03-25T18:11:49.689 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.688+0000 7f7e96ffd640 1 -- 172.21.15.66:0/4243229791 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7e900018b0 con 0x7f7ea013f710 2024-03-25T18:11:49.689 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.688+0000 7f7e96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:49.689 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.688+0000 7f7e96ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:49.689 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.688+0000 7f7e96ffd640 10 monclient(hunting): dump: 2024-03-25T18:11:49.689 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:49.689 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:49.689 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:49.689 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:49.689 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:49.689 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:49.689 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:49.689 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:49.689 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:49.690 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:49.690 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.688+0000 7f7e96ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:49.692 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.692+0000 7f7e96ffd640 1 -- 172.21.15.66:0/4243229791 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2119995223 0 0) 0x7f7ea01dc070 con 0x7f7ea00a98f0 2024-03-25T18:11:49.693 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.692+0000 7f7e96ffd640 1 -- 172.21.15.66:0/4243229791 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e84004eb0 con 0x7f7ea00a98f0 2024-03-25T18:11:49.693 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.692+0000 7f7e96ffd640 1 -- 172.21.15.66:0/4243229791 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7e9c0018a0 con 0x7f7ea00a98f0 2024-03-25T18:11:49.693 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.692+0000 7f7e96ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:49.693 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.692+0000 7f7e96ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:49.693 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.692+0000 7f7e96ffd640 10 monclient(hunting): dump: 2024-03-25T18:11:49.693 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:49.693 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:49.693 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:49.693 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:49.693 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:49.693 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:49.693 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:49.693 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:49.693 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:49.693 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:49.701 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.700+0000 7f7e96ffd640 1 -- 172.21.15.66:0/4243229791 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 128995907 0 0) 0x7f7ea0137f10 con 0x7f7ea013f710 2024-03-25T18:11:49.701 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.700+0000 7f7e96ffd640 1 -- 172.21.15.66:0/4243229791 >> v1:172.21.15.66:6790/0 conn(0x7f7ea01430c0 legacy=0x7f7ea01db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:49.701 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.700+0000 7f7e96ffd640 1 -- 172.21.15.66:0/4243229791 >> v1:172.21.15.66:6789/0 conn(0x7f7ea00a98f0 legacy=0x7f7ea0137800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:49.701 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.700+0000 7f7e96ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:11:49.701 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.700+0000 7f7e96ffd640 1 monclient: found mon.b 2024-03-25T18:11:49.701 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.700+0000 7f7e96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:49.701 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.700+0000 7f7e96ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:11:49.702 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.700+0000 7f7e96ffd640 1 -- 172.21.15.66:0/4243229791 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea01373e0 con 0x7f7ea013f710 2024-03-25T18:11:49.702 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.700+0000 7f7e96ffd640 10 monclient: _finish_auth 0 2024-03-25T18:11:49.702 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.700+0000 7f7e96ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:11:49.702 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.700+0000 7f7e96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:49.702 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.700+0000 7f7ea85a3640 5 monclient: authenticate success, global_id 4576 2024-03-25T18:11:49.702 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.700+0000 7f7ea85a3640 10 monclient: _renew_subs 2024-03-25T18:11:49.702 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.700+0000 7f7ea85a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:11:49.702 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.700+0000 7f7ea85a3640 1 -- 172.21.15.66:0/4243229791 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ea01358e0 con 0x7f7ea013f710 2024-03-25T18:11:49.702 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.700+0000 7f7ea85a3640 10 client.4576.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:49.702 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.700+0000 7f7ea85a3640 10 monclient: _renew_subs 2024-03-25T18:11:49.702 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.700+0000 7f7ea85a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:11:49.702 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.700+0000 7f7ea85a3640 1 -- 172.21.15.66:0/4243229791 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ea0136150 con 0x7f7ea013f710 2024-03-25T18:11:49.702 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.700+0000 7f7ea85a3640 1 librados: init done 2024-03-25T18:11:49.703 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.702+0000 7f7ea85a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:49.703 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.702+0000 7f7ea85a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:49.703 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.702+0000 7f7ea85a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:11:49.703 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.702+0000 7f7ea85a3640 1 -- 172.21.15.66:0/4243229791 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ea00627b0 con 0x7f7ea013f710 2024-03-25T18:11:49.976 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.975+0000 7f7e96ffd640 1 -- 172.21.15.66:0/4243229791 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e900047e0 con 0x7f7ea013f710 2024-03-25T18:11:49.976 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.975+0000 7f7e96ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:49.976 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.975+0000 7f7e96ffd640 1 -- 172.21.15.66:0/4243229791 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7e90004c30 con 0x7f7ea013f710 2024-03-25T18:11:49.976 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.975+0000 7f7e96ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:49.976 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.975+0000 7f7e96ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:49.976 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.975+0000 7f7e96ffd640 10 monclient: dump: 2024-03-25T18:11:49.976 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:49.977 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:49.977 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:49.977 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:49.977 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:49.977 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:49.977 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:49.977 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:49.977 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:49.977 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:49.977 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.975+0000 7f7ea4b3c640 4 set_mon_vals no callback set 2024-03-25T18:11:49.978 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.976+0000 7f7e96ffd640 1 -- 172.21.15.66:0/4243229791 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f7e90004e10 con 0x7f7ea013f710 2024-03-25T18:11:49.978 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.977+0000 7f7ea633f640 10 monclient: get_auth_request con 0x7f7e8406de40 auth_method 0 2024-03-25T18:11:49.978 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.977+0000 7f7e96ffd640 10 client.4576.objecter ms_handle_connect 0x7f7e8406de40 2024-03-25T18:11:49.982 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.981+0000 7f7e96ffd640 1 -- 172.21.15.66:0/4243229791 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f7e90054890 con 0x7f7ea013f710 2024-03-25T18:11:49.982 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.981+0000 7f7e96ffd640 10 client.4576.objecter ms_dispatch 0x7f7ea0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:49.982 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.981+0000 7f7e96ffd640 3 client.4576.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:49.982 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.981+0000 7f7e96ffd640 3 client.4576.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:49.982 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.981+0000 7f7e96ffd640 20 client.4576.objecter dump_active .. 0 homeless 2024-03-25T18:11:49.982 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.982+0000 7f7e96ffd640 1 -- 172.21.15.66:0/4243229791 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7e900503a0 con 0x7f7ea013f710 2024-03-25T18:11:49.982 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.982+0000 7f7e96ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:49.982 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:49.982+0000 7f7e96ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:50.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:50.233+0000 7f7ea85a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:50.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:50.233+0000 7f7ea85a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:50.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:50.233+0000 7f7ea85a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:11:50.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:50.233+0000 7f7ea85a3640 1 -- 172.21.15.66:0/4243229791 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ea0135a70 con 0x7f7ea013f710 2024-03-25T18:11:50.254 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:08:21.234 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:50.259 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:08:21.240 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:50.290 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:08:21.271 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:50.318 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:08:21.276 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:50.323 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:50.322+0000 7f7e96ffd640 1 -- 172.21.15.66:0/4243229791 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f7e900522d0 con 0x7f7ea013f710 2024-03-25T18:11:50.323 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:50.322+0000 7f7e96ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:50.323 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:50.322+0000 7f7e96ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:50.324 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:11:50.324 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:50.326 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:50.324+0000 7f7ea85a3640 10 librados: watch_flush enter 2024-03-25T18:11:50.326 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:50.324+0000 7f7ea85a3640 10 librados: watch_flush exit 2024-03-25T18:11:50.326 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:50.324+0000 7f7ea85a3640 20 client.4576.objecter shutdown clearing up homeless session... 2024-03-25T18:11:50.326 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:50.324+0000 7f7ea85a3640 10 client.4576.objecter successfully canceled tick 2024-03-25T18:11:50.326 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:50.324+0000 7f7ea85a3640 1 -- 172.21.15.66:0/4243229791 >> v1:172.21.15.193:6800/2275676803 conn(0x7f7e8406de40 legacy=0x7f7e84070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:50.327 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:50.324+0000 7f7ea85a3640 10 monclient: shutdown 2024-03-25T18:11:50.327 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:50.324+0000 7f7ea85a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:50.327 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:50.324+0000 7f7ea85a3640 1 -- 172.21.15.66:0/4243229791 >> v1:172.21.15.193:6789/0 conn(0x7f7ea013f710 legacy=0x7f7ea01db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:50.327 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:50.325+0000 7f7ea85a3640 1 -- 172.21.15.66:0/4243229791 shutdown_connections 2024-03-25T18:11:50.327 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:50.325+0000 7f7ea85a3640 1 -- 172.21.15.66:0/4243229791 >> 172.21.15.66:0/4243229791 conn(0x7f7ea00b7ea0 msgr2=0x7f7ea014ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:50.327 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:50.325+0000 7f7ea85a3640 1 -- 172.21.15.66:0/4243229791 shutdown_connections 2024-03-25T18:11:50.327 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:50.325+0000 7f7ea85a3640 1 -- 172.21.15.66:0/4243229791 wait complete. 2024-03-25T18:11:50.327 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:50.325+0000 7f7ea85a3640 1 librados: shutdown 2024-03-25T18:11:50.327 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:08:21.306 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:50.330 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:08:21.289 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:50.332 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:08:21.313 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:50.337 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:50.354 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:08:21.334 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:50.362 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:08:21.321 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:57.144 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:57.262 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.261+0000 7f4ceac25640 10 monclient: get_monmap_and_config 2024-03-25T18:11:57.262 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.261+0000 7f4ceac25640 10 monclient: build_initial_monmap 2024-03-25T18:11:57.262 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.261+0000 7f4ceac25640 10 monclient: monmap: 2024-03-25T18:11:57.262 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:11:57.262 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:57.262 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:11:57.261930+0000 2024-03-25T18:11:57.262 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:11:57.261930+0000 2024-03-25T18:11:57.263 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:57.263 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:57.263 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:11:57.263 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:11:57.263 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:11:57.263 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:57.264 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.262+0000 7f4ceac25640 1 Processor -- start 2024-03-25T18:11:57.264 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.262+0000 7f4ceac25640 1 -- start start 2024-03-25T18:11:57.264 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.262+0000 7f4ceac25640 10 monclient: init 2024-03-25T18:11:57.264 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.262+0000 7f4ceac25640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:57.264 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.262+0000 7f4ceac25640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:11:57.264 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.263+0000 7f4ceac25640 10 monclient(hunting): picked mon.noname-c con 0x7f4ce40a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:11:57.265 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.263+0000 7f4ceac25640 10 monclient(hunting): picked mon.noname-b con 0x7f4ce413f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:11:57.265 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.263+0000 7f4ceac25640 10 monclient(hunting): picked mon.noname-a con 0x7f4ce41430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:11:57.265 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.263+0000 7f4ceac25640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce4142520 con 0x7f4ce41430c0 2024-03-25T18:11:57.265 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.263+0000 7f4ceac25640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce40b28a0 con 0x7f4ce40a98f0 2024-03-25T18:11:57.265 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.263+0000 7f4ceac25640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce40b2a30 con 0x7f4ce413f710 2024-03-25T18:11:57.265 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.263+0000 7f4ceac25640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:57.265 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.263+0000 7f4ceac25640 10 monclient(hunting): authenticate will time out at 7466.322190s 2024-03-25T18:11:57.281 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.281+0000 7f4ce91c2640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f4ce41430c0 0x7f4ce414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:39608/0 (socket says 172.21.15.66:39608) 2024-03-25T18:11:57.281 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.281+0000 7f4ce91c2640 1 -- 172.21.15.66:0/314141813 learned_addr learned my addr 172.21.15.66:0/314141813 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:11:57.300 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.299+0000 7f4ce37fe640 1 -- 172.21.15.66:0/314141813 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846674044 0 0) 0x7f4ce4142520 con 0x7f4ce41430c0 2024-03-25T18:11:57.300 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.299+0000 7f4ce37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:57.300 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.299+0000 7f4ce37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:57.300 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.299+0000 7f4ce37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:57.300 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.299+0000 7f4ce37fe640 10 monclient(hunting): my global_id is 4584 2024-03-25T18:11:57.300 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.300+0000 7f4ce37fe640 1 -- 172.21.15.66:0/314141813 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cd4003520 con 0x7f4ce41430c0 2024-03-25T18:11:57.304 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.304+0000 7f4ce37fe640 1 -- 172.21.15.66:0/314141813 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832558905 0 0) 0x7f4ce40b28a0 con 0x7f4ce40a98f0 2024-03-25T18:11:57.304 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.304+0000 7f4ce37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:57.304 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.304+0000 7f4ce37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:57.305 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.304+0000 7f4ce37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:57.305 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.304+0000 7f4ce37fe640 10 monclient(hunting): my global_id is 4579 2024-03-25T18:11:57.305 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.304+0000 7f4ce37fe640 1 -- 172.21.15.66:0/314141813 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ce4142520 con 0x7f4ce40a98f0 2024-03-25T18:11:57.313 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.312+0000 7f4ce37fe640 1 -- 172.21.15.66:0/314141813 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1861646842 0 0) 0x7f4ce40b2a30 con 0x7f4ce413f710 2024-03-25T18:11:57.313 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.312+0000 7f4ce37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:57.313 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.312+0000 7f4ce37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:57.313 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.312+0000 7f4ce37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:57.313 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.312+0000 7f4ce37fe640 10 monclient(hunting): my global_id is 4580 2024-03-25T18:11:57.313 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.312+0000 7f4ce37fe640 1 -- 172.21.15.66:0/314141813 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ce40b28a0 con 0x7f4ce413f710 2024-03-25T18:11:57.317 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.316+0000 7f4ce37fe640 1 -- 172.21.15.66:0/314141813 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 694498340 0 0) 0x7f4cd4003520 con 0x7f4ce41430c0 2024-03-25T18:11:57.317 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.316+0000 7f4ce37fe640 1 -- 172.21.15.66:0/314141813 >> v1:172.21.15.66:6790/0 conn(0x7f4ce413f710 legacy=0x7f4ce41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:57.317 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.316+0000 7f4ce37fe640 1 -- 172.21.15.66:0/314141813 >> v1:172.21.15.193:6789/0 conn(0x7f4ce40a98f0 legacy=0x7f4ce40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:57.317 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.316+0000 7f4ce37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:11:57.317 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.316+0000 7f4ce37fe640 1 monclient: found mon.noname-a 2024-03-25T18:11:57.317 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.316+0000 7f4ce37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:57.317 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.316+0000 7f4ce37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.66:6789/0 2024-03-25T18:11:57.317 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.316+0000 7f4ce37fe640 1 -- 172.21.15.66:0/314141813 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce40b44c0 con 0x7f4ce41430c0 2024-03-25T18:11:57.318 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.316+0000 7f4ce37fe640 10 monclient: _finish_auth 0 2024-03-25T18:11:57.318 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.316+0000 7f4ce37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:11:57.318 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.316+0000 7f4ce37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:57.318 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.316+0000 7f4ce37fe640 1 -- 172.21.15.66:0/314141813 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f4cdc001a10 con 0x7f4ce41430c0 2024-03-25T18:11:57.318 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.316+0000 7f4ce37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:57.318 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.316+0000 7f4ce37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:57.318 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.316+0000 7f4ce37fe640 10 monclient: dump: 2024-03-25T18:11:57.318 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:57.318 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:57.318 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:57.318 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:57.318 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:57.318 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:57.318 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:57.318 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:57.319 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:57.319 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:57.319 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.316+0000 7f4ceac25640 5 monclient: authenticate success, global_id 4584 2024-03-25T18:11:57.319 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.316+0000 7f4ceac25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:57.362 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.361+0000 7f4ce37fe640 1 -- 172.21.15.66:0/314141813 <== mon.0 v1:172.21.15.66:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4cdc001b70 con 0x7f4ce41430c0 2024-03-25T18:11:57.362 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.361+0000 7f4ce37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:57.362 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.361+0000 7f4ce37fe640 1 -- 172.21.15.66:0/314141813 <== mon.0 v1:172.21.15.66:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f4cdc001f80 con 0x7f4ce41430c0 2024-03-25T18:11:57.362 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.361+0000 7f4ce37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:57.362 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.361+0000 7f4ce37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:57.362 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.361+0000 7f4ce37fe640 10 monclient: dump: 2024-03-25T18:11:57.362 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:57.362 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:57.362 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:57.362 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:57.362 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:57.362 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:57.362 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:57.363 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:57.363 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:57.363 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:57.363 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.361+0000 7f4ceac25640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:57.363 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.361+0000 7f4ceac25640 4 set_mon_vals no callback set 2024-03-25T18:11:57.365 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.362+0000 7f4ceac25640 10 monclient: shutdown 2024-03-25T18:11:57.365 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.362+0000 7f4ceac25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:57.365 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.362+0000 7f4ceac25640 1 -- 172.21.15.66:0/314141813 >> v1:172.21.15.66:6789/0 conn(0x7f4ce41430c0 legacy=0x7f4ce414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:57.365 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.362+0000 7f4ceac25640 1 -- 172.21.15.66:0/314141813 shutdown_connections 2024-03-25T18:11:57.365 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.362+0000 7f4ceac25640 1 -- 172.21.15.66:0/314141813 >> 172.21.15.66:0/314141813 conn(0x7f4ce40b7ea0 msgr2=0x7f4ce40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:57.365 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.362+0000 7f4ceac25640 1 -- 172.21.15.66:0/314141813 shutdown_connections 2024-03-25T18:11:57.365 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.362+0000 7f4ceac25640 1 -- 172.21.15.66:0/314141813 wait complete. 2024-03-25T18:11:57.365 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.363+0000 7f4ceac25640 10 monclient: build_initial_monmap 2024-03-25T18:11:57.365 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.363+0000 7f4ceac25640 10 monclient: monmap: 2024-03-25T18:11:57.366 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:11:57.366 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:57.366 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:11:57.366 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:11:57.366 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:57.366 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:11:57.366 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:11:57.366 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:11:57.366 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:11:57.366 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:57.366 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.363+0000 7f4ceac25640 1 librados: starting msgr at 2024-03-25T18:11:57.366 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.363+0000 7f4ceac25640 1 librados: starting objecter 2024-03-25T18:11:57.366 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.363+0000 7f4ceac25640 1 Processor -- start 2024-03-25T18:11:57.366 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.363+0000 7f4ceac25640 1 -- start start 2024-03-25T18:11:57.366 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.363+0000 7f4ceac25640 1 librados: setting wanted keys 2024-03-25T18:11:57.366 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.363+0000 7f4ceac25640 1 librados: calling monclient init 2024-03-25T18:11:57.366 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.363+0000 7f4ceac25640 10 monclient: init 2024-03-25T18:11:57.367 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.364+0000 7f4ceac25640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:57.367 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.364+0000 7f4ceac25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:11:57.367 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.364+0000 7f4ceac25640 10 monclient(hunting): picked mon.noname-a con 0x7f4ce40a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:11:57.367 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.364+0000 7f4ceac25640 10 monclient(hunting): picked mon.noname-b con 0x7f4ce413f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:11:57.367 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.364+0000 7f4ceac25640 10 monclient(hunting): picked mon.noname-c con 0x7f4ce41430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:11:57.367 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.364+0000 7f4ceac25640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce413b410 con 0x7f4ce40a98f0 2024-03-25T18:11:57.367 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.364+0000 7f4ceac25640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce413b010 con 0x7f4ce413f710 2024-03-25T18:11:57.367 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.364+0000 7f4ceac25640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce4135230 con 0x7f4ce41430c0 2024-03-25T18:11:57.367 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.364+0000 7f4ceac25640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:57.367 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.364+0000 7f4ceac25640 10 monclient(hunting): authenticate will time out at 7466.423825s 2024-03-25T18:11:57.377 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.377+0000 7f4ce3fff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f4ce413f710 0x7f4ce41dabe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:52886/0 (socket says 172.21.15.66:52886) 2024-03-25T18:11:57.377 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.377+0000 7f4ce3fff640 1 -- 172.21.15.66:0/152882702 learned_addr learned my addr 172.21.15.66:0/152882702 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:11:57.381 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.380+0000 7f4ce17fa640 10 client.?.objecter ms_handle_connect 0x7f4ce40a98f0 2024-03-25T18:11:57.381 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.380+0000 7f4ce17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:57.381 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.380+0000 7f4ce17fa640 10 client.?.objecter ms_handle_connect 0x7f4ce413f710 2024-03-25T18:11:57.381 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.380+0000 7f4ce17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:57.384 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.384+0000 7f4ce17fa640 10 client.?.objecter ms_handle_connect 0x7f4ce41430c0 2024-03-25T18:11:57.384 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.384+0000 7f4ce17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:57.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.396+0000 7f4ce17fa640 1 -- 172.21.15.66:0/152882702 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 295650160 0 0) 0x7f4ce413b010 con 0x7f4ce413f710 2024-03-25T18:11:57.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.396+0000 7f4ce17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:11:57.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.396+0000 7f4ce17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:57.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.396+0000 7f4ce17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:57.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.396+0000 7f4ce17fa640 10 monclient(hunting): my global_id is 4582 2024-03-25T18:11:57.397 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.396+0000 7f4ce17fa640 1 -- 172.21.15.66:0/152882702 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cc8004eb0 con 0x7f4ce413f710 2024-03-25T18:11:57.402 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.401+0000 7f4ce17fa640 1 -- 172.21.15.66:0/152882702 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3986645557 0 0) 0x7f4ce413b410 con 0x7f4ce40a98f0 2024-03-25T18:11:57.402 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.401+0000 7f4ce17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:11:57.402 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.401+0000 7f4ce17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:57.402 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.401+0000 7f4ce17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:57.402 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.401+0000 7f4ce17fa640 10 monclient(hunting): my global_id is 4587 2024-03-25T18:11:57.402 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.401+0000 7f4ce17fa640 1 -- 172.21.15.66:0/152882702 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ce413b010 con 0x7f4ce40a98f0 2024-03-25T18:11:57.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.402+0000 7f4ce17fa640 1 -- 172.21.15.66:0/152882702 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1182703444 0 0) 0x7f4ce4135230 con 0x7f4ce41430c0 2024-03-25T18:11:57.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.402+0000 7f4ce17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:11:57.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.402+0000 7f4ce17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:57.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.402+0000 7f4ce17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:57.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.402+0000 7f4ce17fa640 10 monclient(hunting): my global_id is 4583 2024-03-25T18:11:57.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.402+0000 7f4ce17fa640 1 -- 172.21.15.66:0/152882702 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ce413b410 con 0x7f4ce41430c0 2024-03-25T18:11:57.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.404+0000 7f4ce17fa640 1 -- 172.21.15.66:0/152882702 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3694669960 0 0) 0x7f4cc8004eb0 con 0x7f4ce413f710 2024-03-25T18:11:57.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.404+0000 7f4ce17fa640 1 -- 172.21.15.66:0/152882702 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ce4135230 con 0x7f4ce413f710 2024-03-25T18:11:57.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.404+0000 7f4ce17fa640 1 -- 172.21.15.66:0/152882702 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f4cd0001a10 con 0x7f4ce413f710 2024-03-25T18:11:57.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.404+0000 7f4ce17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:57.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.404+0000 7f4ce17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:57.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.404+0000 7f4ce17fa640 10 monclient(hunting): dump: 2024-03-25T18:11:57.405 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:57.405 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:57.406 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:57.406 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:57.406 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:57.406 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:57.406 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:57.406 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:57.406 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:57.406 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:57.406 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.404+0000 7f4ce17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:57.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.412+0000 7f4ce17fa640 1 -- 172.21.15.66:0/152882702 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2254975801 0 0) 0x7f4ce4135230 con 0x7f4ce413f710 2024-03-25T18:11:57.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.412+0000 7f4ce17fa640 1 -- 172.21.15.66:0/152882702 >> v1:172.21.15.66:6790/0 conn(0x7f4ce41430c0 legacy=0x7f4ce41db2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:57.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.412+0000 7f4ce17fa640 1 -- 172.21.15.66:0/152882702 >> v1:172.21.15.66:6789/0 conn(0x7f4ce40a98f0 legacy=0x7f4ce4130ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:57.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.412+0000 7f4ce17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:11:57.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.412+0000 7f4ce17fa640 1 monclient: found mon.b 2024-03-25T18:11:57.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.412+0000 7f4ce17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:57.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.412+0000 7f4ce17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:11:57.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.412+0000 7f4ce17fa640 1 -- 172.21.15.66:0/152882702 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce41353c0 con 0x7f4ce413f710 2024-03-25T18:11:57.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.412+0000 7f4ce17fa640 10 monclient: _finish_auth 0 2024-03-25T18:11:57.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.412+0000 7f4ce17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:11:57.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.412+0000 7f4ce17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:57.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.412+0000 7f4ceac25640 5 monclient: authenticate success, global_id 4582 2024-03-25T18:11:57.414 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.412+0000 7f4ceac25640 10 monclient: _renew_subs 2024-03-25T18:11:57.414 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.412+0000 7f4ceac25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:11:57.414 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.412+0000 7f4ceac25640 1 -- 172.21.15.66:0/152882702 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ce40b78e0 con 0x7f4ce413f710 2024-03-25T18:11:57.414 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.412+0000 7f4ceac25640 10 client.4582.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:57.414 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.412+0000 7f4ceac25640 10 monclient: _renew_subs 2024-03-25T18:11:57.414 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.412+0000 7f4ceac25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:11:57.414 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.412+0000 7f4ceac25640 1 -- 172.21.15.66:0/152882702 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ce41315f0 con 0x7f4ce413f710 2024-03-25T18:11:57.414 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.412+0000 7f4ceac25640 1 librados: init done 2024-03-25T18:11:57.415 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.413+0000 7f4ceac25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:57.415 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.413+0000 7f4ceac25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:57.415 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.413+0000 7f4ceac25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:11:57.415 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.413+0000 7f4ceac25640 1 -- 172.21.15.66:0/152882702 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ce40627b0 con 0x7f4ce413f710 2024-03-25T18:11:57.427 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.426+0000 7f4ce17fa640 1 -- 172.21.15.66:0/152882702 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4cd0001c70 con 0x7f4ce413f710 2024-03-25T18:11:57.427 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.426+0000 7f4ce17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:57.427 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.427+0000 7f4ce2ffd640 4 set_mon_vals no callback set 2024-03-25T18:11:57.682 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.682+0000 7f4ce17fa640 1 -- 172.21.15.66:0/152882702 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f4cd0003a60 con 0x7f4ce413f710 2024-03-25T18:11:57.683 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.682+0000 7f4ce17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:57.683 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.682+0000 7f4ce17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:57.683 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.682+0000 7f4ce17fa640 10 monclient: dump: 2024-03-25T18:11:57.683 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:11:57.683 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:11:57.683 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:57.683 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:11:57.683 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:57.683 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:11:57.683 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:11:57.683 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:11:57.683 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:11:57.684 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:11:57.684 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.683+0000 7f4ce17fa640 1 -- 172.21.15.66:0/152882702 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f4cd0003c40 con 0x7f4ce413f710 2024-03-25T18:11:57.684 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.684+0000 7f4ce89c1640 10 monclient: get_auth_request con 0x7f4cc806ddf0 auth_method 0 2024-03-25T18:11:57.685 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.684+0000 7f4ce17fa640 10 client.4582.objecter ms_handle_connect 0x7f4cc806ddf0 2024-03-25T18:11:57.686 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.686+0000 7f4ce17fa640 1 -- 172.21.15.66:0/152882702 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f4cd0054920 con 0x7f4ce413f710 2024-03-25T18:11:57.686 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.686+0000 7f4ce17fa640 10 client.4582.objecter ms_dispatch 0x7f4ce4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:57.687 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.686+0000 7f4ce17fa640 3 client.4582.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:57.687 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.686+0000 7f4ce17fa640 3 client.4582.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:57.687 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.686+0000 7f4ce17fa640 20 client.4582.objecter dump_active .. 0 homeless 2024-03-25T18:11:57.690 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.689+0000 7f4ce17fa640 1 -- 172.21.15.66:0/152882702 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4cd0050260 con 0x7f4ce413f710 2024-03-25T18:11:57.690 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.689+0000 7f4ce17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:57.690 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.689+0000 7f4ce17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:57.950 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.949+0000 7f4ceac25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:57.950 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.949+0000 7f4ceac25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:57.950 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.949+0000 7f4ceac25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:11:57.950 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.949+0000 7f4ceac25640 1 -- 172.21.15.66:0/152882702 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ce413b5c0 con 0x7f4ce413f710 2024-03-25T18:11:57.982 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.981+0000 7f4ce17fa640 1 -- 172.21.15.66:0/152882702 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f4ce413b5c0 con 0x7f4ce413f710 2024-03-25T18:11:57.982 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.981+0000 7f4ce17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:57.982 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.981+0000 7f4ce17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:57.982 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:11:57.982 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:57.985 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.984+0000 7f4ceac25640 10 librados: watch_flush enter 2024-03-25T18:11:57.985 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.984+0000 7f4ceac25640 10 librados: watch_flush exit 2024-03-25T18:11:57.985 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.984+0000 7f4ceac25640 20 client.4582.objecter shutdown clearing up homeless session... 2024-03-25T18:11:57.985 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.984+0000 7f4ceac25640 10 client.4582.objecter successfully canceled tick 2024-03-25T18:11:57.985 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.984+0000 7f4ceac25640 1 -- 172.21.15.66:0/152882702 >> v1:172.21.15.193:6800/2275676803 conn(0x7f4cc806ddf0 legacy=0x7f4cc80702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:57.985 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.984+0000 7f4ceac25640 10 monclient: shutdown 2024-03-25T18:11:57.985 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.984+0000 7f4ceac25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:57.985 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.984+0000 7f4ceac25640 1 -- 172.21.15.66:0/152882702 >> v1:172.21.15.193:6789/0 conn(0x7f4ce413f710 legacy=0x7f4ce41dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:57.985 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.984+0000 7f4ceac25640 1 -- 172.21.15.66:0/152882702 shutdown_connections 2024-03-25T18:11:57.986 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.984+0000 7f4ceac25640 1 -- 172.21.15.66:0/152882702 >> 172.21.15.66:0/152882702 conn(0x7f4ce40b7ea0 msgr2=0x7f4ce4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:57.986 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.984+0000 7f4ceac25640 1 -- 172.21.15.66:0/152882702 shutdown_connections 2024-03-25T18:11:57.986 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.984+0000 7f4ceac25640 1 -- 172.21.15.66:0/152882702 wait complete. 2024-03-25T18:11:57.986 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:11:57.984+0000 7f4ceac25640 1 librados: shutdown 2024-03-25T18:11:57.997 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:04.804 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:04.922 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.921+0000 7f6e22d4d640 10 monclient: get_monmap_and_config 2024-03-25T18:12:04.922 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.921+0000 7f6e22d4d640 10 monclient: build_initial_monmap 2024-03-25T18:12:04.922 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.921+0000 7f6e22d4d640 10 monclient: monmap: 2024-03-25T18:12:04.923 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:12:04.923 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:04.923 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:12:04.922169+0000 2024-03-25T18:12:04.923 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:12:04.922169+0000 2024-03-25T18:12:04.923 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:04.923 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:04.923 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:12:04.923 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:12:04.923 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:12:04.923 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:04.924 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.922+0000 7f6e22d4d640 1 Processor -- start 2024-03-25T18:12:04.924 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.922+0000 7f6e22d4d640 1 -- start start 2024-03-25T18:12:04.924 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.922+0000 7f6e22d4d640 10 monclient: init 2024-03-25T18:12:04.924 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.923+0000 7f6e22d4d640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:04.924 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.923+0000 7f6e22d4d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:12:04.925 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.923+0000 7f6e22d4d640 10 monclient(hunting): picked mon.noname-a con 0x7f6e1c0a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:12:04.925 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.923+0000 7f6e22d4d640 10 monclient(hunting): picked mon.noname-b con 0x7f6e1c13f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:12:04.925 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.923+0000 7f6e22d4d640 10 monclient(hunting): picked mon.noname-c con 0x7f6e1c1430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:12:04.925 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.923+0000 7f6e22d4d640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e1c142520 con 0x7f6e1c0a98f0 2024-03-25T18:12:04.925 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.923+0000 7f6e22d4d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e1c0b28a0 con 0x7f6e1c1430c0 2024-03-25T18:12:04.925 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.923+0000 7f6e22d4d640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e1c0b2a30 con 0x7f6e1c13f710 2024-03-25T18:12:04.925 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.923+0000 7f6e22d4d640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:04.925 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.923+0000 7f6e22d4d640 10 monclient(hunting): authenticate will time out at 7473.982856s 2024-03-25T18:12:04.937 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.937+0000 7f6e20ae9640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f6e1c0a98f0 0x7f6e1c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:37146/0 (socket says 172.21.15.66:37146) 2024-03-25T18:12:04.938 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.937+0000 7f6e20ae9640 1 -- 172.21.15.66:0/4211062068 learned_addr learned my addr 172.21.15.66:0/4211062068 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:12:04.954 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.953+0000 7f6e1b7fe640 1 -- 172.21.15.66:0/4211062068 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4293205961 0 0) 0x7f6e1c0b28a0 con 0x7f6e1c1430c0 2024-03-25T18:12:04.954 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.953+0000 7f6e1b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:12:04.954 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.953+0000 7f6e1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:04.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.953+0000 7f6e1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:04.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.953+0000 7f6e1b7fe640 10 monclient(hunting): my global_id is 4585 2024-03-25T18:12:04.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.953+0000 7f6e1b7fe640 1 -- 172.21.15.66:0/4211062068 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e0c003520 con 0x7f6e1c1430c0 2024-03-25T18:12:04.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.954+0000 7f6e1b7fe640 1 -- 172.21.15.66:0/4211062068 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2735300968 0 0) 0x7f6e1c142520 con 0x7f6e1c0a98f0 2024-03-25T18:12:04.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.954+0000 7f6e1b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:12:04.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.954+0000 7f6e1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:04.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.954+0000 7f6e1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:04.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.954+0000 7f6e1b7fe640 10 monclient(hunting): my global_id is 4590 2024-03-25T18:12:04.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.954+0000 7f6e1b7fe640 1 -- 172.21.15.66:0/4211062068 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e1c0b28a0 con 0x7f6e1c0a98f0 2024-03-25T18:12:04.971 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.970+0000 7f6e1b7fe640 1 -- 172.21.15.66:0/4211062068 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 728969643 0 0) 0x7f6e0c003520 con 0x7f6e1c1430c0 2024-03-25T18:12:04.971 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.970+0000 7f6e1b7fe640 1 -- 172.21.15.66:0/4211062068 >> v1:172.21.15.66:6790/0 conn(0x7f6e1c13f710 legacy=0x7f6e1c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:04.971 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.970+0000 7f6e1b7fe640 1 -- 172.21.15.66:0/4211062068 >> v1:172.21.15.66:6789/0 conn(0x7f6e1c0a98f0 legacy=0x7f6e1c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:04.971 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.970+0000 7f6e1b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:12:04.971 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.970+0000 7f6e1b7fe640 1 monclient: found mon.noname-c 2024-03-25T18:12:04.971 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.970+0000 7f6e1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:04.971 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.970+0000 7f6e1b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:12:04.972 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.970+0000 7f6e1b7fe640 1 -- 172.21.15.66:0/4211062068 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1c0b44c0 con 0x7f6e1c1430c0 2024-03-25T18:12:04.972 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.970+0000 7f6e1b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:12:04.972 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.970+0000 7f6e1b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:12:04.972 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.970+0000 7f6e1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:04.972 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.970+0000 7f6e1b7fe640 1 -- 172.21.15.66:0/4211062068 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f6e14001a10 con 0x7f6e1c1430c0 2024-03-25T18:12:04.972 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.970+0000 7f6e22d4d640 5 monclient: authenticate success, global_id 4585 2024-03-25T18:12:04.972 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.970+0000 7f6e22d4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:04.972 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.970+0000 7f6e1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:04.972 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.970+0000 7f6e1b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:04.972 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.970+0000 7f6e1b7fe640 10 monclient: dump: 2024-03-25T18:12:04.972 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:04.972 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:04.972 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:04.972 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:04.972 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:04.972 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:04.972 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:04.973 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:04.973 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:04.973 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:04.973 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.970+0000 7f6e22d4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:04.985 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.985+0000 7f6e1b7fe640 1 -- 172.21.15.66:0/4211062068 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e14001b70 con 0x7f6e1c1430c0 2024-03-25T18:12:04.985 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.985+0000 7f6e1b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:04.985 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.985+0000 7f6e1b7fe640 1 -- 172.21.15.66:0/4211062068 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f6e14001f80 con 0x7f6e1c1430c0 2024-03-25T18:12:04.985 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.985+0000 7f6e1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:04.985 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.985+0000 7f6e1b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:04.986 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.985+0000 7f6e1b7fe640 10 monclient: dump: 2024-03-25T18:12:04.986 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:04.986 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:04.986 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:04.986 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:04.986 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:04.986 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:04.986 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:04.986 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:04.986 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:04.986 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:04.986 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.985+0000 7f6e22d4d640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:04.986 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.985+0000 7f6e22d4d640 4 set_mon_vals no callback set 2024-03-25T18:12:04.989 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.985+0000 7f6e22d4d640 10 monclient: shutdown 2024-03-25T18:12:04.989 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.985+0000 7f6e22d4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:04.989 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.985+0000 7f6e22d4d640 1 -- 172.21.15.66:0/4211062068 >> v1:172.21.15.193:6789/0 conn(0x7f6e1c1430c0 legacy=0x7f6e1c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:04.989 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.986+0000 7f6e22d4d640 1 -- 172.21.15.66:0/4211062068 shutdown_connections 2024-03-25T18:12:04.989 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.986+0000 7f6e22d4d640 1 -- 172.21.15.66:0/4211062068 >> 172.21.15.66:0/4211062068 conn(0x7f6e1c0b7ea0 msgr2=0x7f6e1c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:04.989 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.986+0000 7f6e22d4d640 1 -- 172.21.15.66:0/4211062068 shutdown_connections 2024-03-25T18:12:04.989 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.986+0000 7f6e22d4d640 1 -- 172.21.15.66:0/4211062068 wait complete. 2024-03-25T18:12:04.989 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.986+0000 7f6e22d4d640 10 monclient: build_initial_monmap 2024-03-25T18:12:04.989 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.986+0000 7f6e22d4d640 10 monclient: monmap: 2024-03-25T18:12:04.989 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:12:04.990 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:04.990 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:12:04.990 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:12:04.990 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:04.990 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:12:04.990 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:12:04.990 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:12:04.990 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:12:04.990 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:04.990 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.987+0000 7f6e22d4d640 1 librados: starting msgr at 2024-03-25T18:12:04.990 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.987+0000 7f6e22d4d640 1 librados: starting objecter 2024-03-25T18:12:04.990 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.987+0000 7f6e22d4d640 1 Processor -- start 2024-03-25T18:12:04.990 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.987+0000 7f6e22d4d640 1 -- start start 2024-03-25T18:12:04.991 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.987+0000 7f6e22d4d640 1 librados: setting wanted keys 2024-03-25T18:12:04.991 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.987+0000 7f6e22d4d640 1 librados: calling monclient init 2024-03-25T18:12:04.991 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.987+0000 7f6e22d4d640 10 monclient: init 2024-03-25T18:12:04.991 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.987+0000 7f6e22d4d640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:04.991 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.987+0000 7f6e22d4d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:12:04.991 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.988+0000 7f6e22d4d640 10 monclient(hunting): picked mon.noname-b con 0x7f6e1c0a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:12:04.991 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.988+0000 7f6e22d4d640 10 monclient(hunting): picked mon.noname-a con 0x7f6e1c13f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:12:04.991 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.988+0000 7f6e22d4d640 10 monclient(hunting): picked mon.noname-c con 0x7f6e1c1430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:12:04.991 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.988+0000 7f6e22d4d640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e1c1daca0 con 0x7f6e1c13f710 2024-03-25T18:12:04.991 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.988+0000 7f6e22d4d640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e1c138ab0 con 0x7f6e1c0a98f0 2024-03-25T18:12:04.991 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.988+0000 7f6e22d4d640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e1c0aba90 con 0x7f6e1c1430c0 2024-03-25T18:12:04.991 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.988+0000 7f6e22d4d640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:04.991 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.988+0000 7f6e22d4d640 10 monclient(hunting): authenticate will time out at 7474.047182s 2024-03-25T18:12:04.998 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.998+0000 7f6e20ae9640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f6e1c0a98f0 0x7f6e1c1383a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:40522/0 (socket says 172.21.15.66:40522) 2024-03-25T18:12:04.998 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.998+0000 7f6e20ae9640 1 -- 172.21.15.66:0/3024660583 learned_addr learned my addr 172.21.15.66:0/3024660583 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:12:05.000 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.999+0000 7f6e197fa640 10 client.?.objecter ms_handle_connect 0x7f6e1c0a98f0 2024-03-25T18:12:05.000 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:04.999+0000 7f6e197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:05.000 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.000+0000 7f6e197fa640 10 client.?.objecter ms_handle_connect 0x7f6e1c13f710 2024-03-25T18:12:05.000 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.000+0000 7f6e197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:05.005 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.004+0000 7f6e197fa640 10 client.?.objecter ms_handle_connect 0x7f6e1c1430c0 2024-03-25T18:12:05.005 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.004+0000 7f6e197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:05.008 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.008+0000 7f6e197fa640 1 -- 172.21.15.66:0/3024660583 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3430695985 0 0) 0x7f6e1c1daca0 con 0x7f6e1c13f710 2024-03-25T18:12:05.008 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.008+0000 7f6e197fa640 10 monclient(hunting): _negotiate 2024-03-25T18:12:05.008 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.008+0000 7f6e197fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:05.008 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.008+0000 7f6e197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:05.009 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.008+0000 7f6e197fa640 10 monclient(hunting): my global_id is 4593 2024-03-25T18:12:05.009 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.008+0000 7f6e197fa640 1 -- 172.21.15.66:0/3024660583 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e00004eb0 con 0x7f6e1c13f710 2024-03-25T18:12:05.011 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.010+0000 7f6e197fa640 1 -- 172.21.15.66:0/3024660583 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 24225907 0 0) 0x7f6e1c138ab0 con 0x7f6e1c0a98f0 2024-03-25T18:12:05.011 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.010+0000 7f6e197fa640 10 monclient(hunting): _negotiate 2024-03-25T18:12:05.011 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.010+0000 7f6e197fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:05.011 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.010+0000 7f6e197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:05.011 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.010+0000 7f6e197fa640 10 monclient(hunting): my global_id is 4588 2024-03-25T18:12:05.011 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.010+0000 7f6e197fa640 1 -- 172.21.15.66:0/3024660583 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e1c1daca0 con 0x7f6e1c0a98f0 2024-03-25T18:12:05.019 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.018+0000 7f6e197fa640 1 -- 172.21.15.66:0/3024660583 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2144906522 0 0) 0x7f6e1c1daca0 con 0x7f6e1c0a98f0 2024-03-25T18:12:05.019 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.018+0000 7f6e197fa640 1 -- 172.21.15.66:0/3024660583 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e1c138ab0 con 0x7f6e1c0a98f0 2024-03-25T18:12:05.019 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.018+0000 7f6e197fa640 1 -- 172.21.15.66:0/3024660583 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f6e100019f0 con 0x7f6e1c0a98f0 2024-03-25T18:12:05.019 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.018+0000 7f6e197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:05.019 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.018+0000 7f6e197fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:05.019 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.018+0000 7f6e197fa640 10 monclient(hunting): dump: 2024-03-25T18:12:05.019 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:05.019 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:05.019 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:05.019 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:05.019 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:05.019 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:05.019 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:05.019 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:05.020 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:05.020 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:05.020 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.018+0000 7f6e197fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:05.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.023+0000 7f6e197fa640 1 -- 172.21.15.66:0/3024660583 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3803021564 0 0) 0x7f6e1c0aba90 con 0x7f6e1c1430c0 2024-03-25T18:12:05.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.023+0000 7f6e197fa640 10 monclient(hunting): _negotiate 2024-03-25T18:12:05.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.023+0000 7f6e197fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:05.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.023+0000 7f6e197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:05.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.023+0000 7f6e197fa640 10 monclient(hunting): my global_id is 4589 2024-03-25T18:12:05.023 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.023+0000 7f6e197fa640 1 -- 172.21.15.66:0/3024660583 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e1c1daca0 con 0x7f6e1c1430c0 2024-03-25T18:12:05.024 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.023+0000 7f6e197fa640 1 -- 172.21.15.66:0/3024660583 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2229747547 0 0) 0x7f6e00004eb0 con 0x7f6e1c13f710 2024-03-25T18:12:05.024 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.024+0000 7f6e197fa640 1 -- 172.21.15.66:0/3024660583 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e1c0aba90 con 0x7f6e1c13f710 2024-03-25T18:12:05.024 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.024+0000 7f6e197fa640 1 -- 172.21.15.66:0/3024660583 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f6e08001930 con 0x7f6e1c13f710 2024-03-25T18:12:05.025 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.024+0000 7f6e197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:05.025 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.024+0000 7f6e197fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:05.025 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.024+0000 7f6e197fa640 10 monclient(hunting): dump: 2024-03-25T18:12:05.025 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:05.025 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:05.025 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:05.025 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:05.025 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:05.025 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:05.025 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:05.025 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:05.025 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:05.025 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:05.035 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.033+0000 7f6e197fa640 1 -- 172.21.15.66:0/3024660583 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 721117965 0 0) 0x7f6e1c138ab0 con 0x7f6e1c0a98f0 2024-03-25T18:12:05.035 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.033+0000 7f6e197fa640 1 -- 172.21.15.66:0/3024660583 >> v1:172.21.15.66:6790/0 conn(0x7f6e1c1430c0 legacy=0x7f6e1c1d47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:05.035 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.033+0000 7f6e197fa640 1 -- 172.21.15.66:0/3024660583 >> v1:172.21.15.66:6789/0 conn(0x7f6e1c13f710 legacy=0x7f6e1c1db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:05.035 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.033+0000 7f6e197fa640 10 monclient: _finish_hunting 0 2024-03-25T18:12:05.035 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.033+0000 7f6e197fa640 1 monclient: found mon.b 2024-03-25T18:12:05.035 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.033+0000 7f6e197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:05.035 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.033+0000 7f6e197fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:12:05.035 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.034+0000 7f6e197fa640 1 -- 172.21.15.66:0/3024660583 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1c1d41e0 con 0x7f6e1c0a98f0 2024-03-25T18:12:05.035 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.034+0000 7f6e197fa640 10 monclient: _finish_auth 0 2024-03-25T18:12:05.035 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.034+0000 7f6e197fa640 10 monclient: _check_auth_tickets 2024-03-25T18:12:05.035 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.034+0000 7f6e197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:05.036 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.034+0000 7f6e22d4d640 5 monclient: authenticate success, global_id 4588 2024-03-25T18:12:05.036 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.034+0000 7f6e22d4d640 10 monclient: _renew_subs 2024-03-25T18:12:05.036 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.034+0000 7f6e22d4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:12:05.036 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.034+0000 7f6e22d4d640 1 -- 172.21.15.66:0/3024660583 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e1c1db840 con 0x7f6e1c0a98f0 2024-03-25T18:12:05.036 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.034+0000 7f6e22d4d640 10 client.4588.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:05.036 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.034+0000 7f6e22d4d640 10 monclient: _renew_subs 2024-03-25T18:12:05.036 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.034+0000 7f6e22d4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:12:05.036 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.034+0000 7f6e22d4d640 1 -- 172.21.15.66:0/3024660583 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e1c138f00 con 0x7f6e1c0a98f0 2024-03-25T18:12:05.036 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.034+0000 7f6e22d4d640 1 librados: init done 2024-03-25T18:12:05.037 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.035+0000 7f6e22d4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:05.037 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.035+0000 7f6e22d4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:05.037 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.035+0000 7f6e22d4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:12:05.037 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.035+0000 7f6e22d4d640 1 -- 172.21.15.66:0/3024660583 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e1c0627b0 con 0x7f6e1c0a98f0 2024-03-25T18:12:05.052 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.051+0000 7f6e197fa640 1 -- 172.21.15.66:0/3024660583 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e100047e0 con 0x7f6e1c0a98f0 2024-03-25T18:12:05.052 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.051+0000 7f6e197fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:05.052 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.051+0000 7f6e197fa640 1 -- 172.21.15.66:0/3024660583 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f6e10004af0 con 0x7f6e1c0a98f0 2024-03-25T18:12:05.052 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.051+0000 7f6e197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:05.052 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.051+0000 7f6e197fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:05.052 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.051+0000 7f6e1affd640 4 set_mon_vals no callback set 2024-03-25T18:12:05.052 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.051+0000 7f6e197fa640 10 monclient: dump: 2024-03-25T18:12:05.052 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:05.052 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:05.052 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:05.052 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:05.053 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:05.053 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:05.053 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:05.053 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:05.053 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:05.053 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:05.054 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.052+0000 7f6e197fa640 1 -- 172.21.15.66:0/3024660583 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f6e1001aac0 con 0x7f6e1c0a98f0 2024-03-25T18:12:05.054 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.053+0000 7f6e1bfff640 10 monclient: get_auth_request con 0x7f6e0006df70 auth_method 0 2024-03-25T18:12:05.054 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.053+0000 7f6e197fa640 10 client.4588.objecter ms_handle_connect 0x7f6e0006df70 2024-03-25T18:12:05.055 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.055+0000 7f6e197fa640 1 -- 172.21.15.66:0/3024660583 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f6e10054840 con 0x7f6e1c0a98f0 2024-03-25T18:12:05.055 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.055+0000 7f6e197fa640 10 client.4588.objecter ms_dispatch 0x7f6e1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:05.056 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.055+0000 7f6e197fa640 3 client.4588.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:05.056 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.055+0000 7f6e197fa640 3 client.4588.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:05.056 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.055+0000 7f6e197fa640 20 client.4588.objecter dump_active .. 0 homeless 2024-03-25T18:12:05.313 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.312+0000 7f6e197fa640 1 -- 172.21.15.66:0/3024660583 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6e10050180 con 0x7f6e1c0a98f0 2024-03-25T18:12:05.313 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.312+0000 7f6e197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:05.313 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.312+0000 7f6e197fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:05.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.571+0000 7f6e22d4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:05.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.571+0000 7f6e22d4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:05.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.571+0000 7f6e22d4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:12:05.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.571+0000 7f6e22d4d640 1 -- 172.21.15.66:0/3024660583 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e1c1d4370 con 0x7f6e1c0a98f0 2024-03-25T18:12:05.597 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.596+0000 7f6e197fa640 1 -- 172.21.15.66:0/3024660583 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f6e10052280 con 0x7f6e1c0a98f0 2024-03-25T18:12:05.597 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.596+0000 7f6e197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:05.597 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.596+0000 7f6e197fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:05.597 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:12:05.597 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:05.599 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.598+0000 7f6e22d4d640 10 librados: watch_flush enter 2024-03-25T18:12:05.599 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.598+0000 7f6e22d4d640 10 librados: watch_flush exit 2024-03-25T18:12:05.599 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.598+0000 7f6e22d4d640 20 client.4588.objecter shutdown clearing up homeless session... 2024-03-25T18:12:05.599 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.598+0000 7f6e22d4d640 10 client.4588.objecter successfully canceled tick 2024-03-25T18:12:05.600 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.598+0000 7f6e22d4d640 1 -- 172.21.15.66:0/3024660583 >> v1:172.21.15.193:6800/2275676803 conn(0x7f6e0006df70 legacy=0x7f6e00070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:05.600 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.598+0000 7f6e22d4d640 10 monclient: shutdown 2024-03-25T18:12:05.600 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.598+0000 7f6e22d4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:05.600 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.598+0000 7f6e22d4d640 1 -- 172.21.15.66:0/3024660583 >> v1:172.21.15.193:6789/0 conn(0x7f6e1c0a98f0 legacy=0x7f6e1c1383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:05.600 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.598+0000 7f6e22d4d640 1 -- 172.21.15.66:0/3024660583 shutdown_connections 2024-03-25T18:12:05.600 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.598+0000 7f6e22d4d640 1 -- 172.21.15.66:0/3024660583 >> 172.21.15.66:0/3024660583 conn(0x7f6e1c0b7ea0 msgr2=0x7f6e1c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:05.600 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.598+0000 7f6e22d4d640 1 -- 172.21.15.66:0/3024660583 shutdown_connections 2024-03-25T18:12:05.600 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.598+0000 7f6e22d4d640 1 -- 172.21.15.66:0/3024660583 wait complete. 2024-03-25T18:12:05.600 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:05.598+0000 7f6e22d4d640 1 librados: shutdown 2024-03-25T18:12:05.611 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:10.101 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:08:41.082 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:10.113 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:08:41.093 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:10.147 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:08:41.128 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:10.172 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:08:41.131 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:10.175 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:08:41.155 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:10.179 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:08:41.137 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:10.187 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:08:41.167 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:10.200 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:08:41.158 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:10.225 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:08:41.205 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:12.418 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:12.523 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.522+0000 7fe5c58f6640 10 monclient: get_monmap_and_config 2024-03-25T18:12:12.523 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.522+0000 7fe5c58f6640 10 monclient: build_initial_monmap 2024-03-25T18:12:12.523 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.522+0000 7fe5c58f6640 10 monclient: monmap: 2024-03-25T18:12:12.523 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:12:12.523 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:12.523 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:12:12.523258+0000 2024-03-25T18:12:12.523 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:12:12.523258+0000 2024-03-25T18:12:12.523 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:12.524 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:12.524 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:12:12.524 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:12:12.524 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:12:12.524 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:12.525 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.523+0000 7fe5c58f6640 1 Processor -- start 2024-03-25T18:12:12.525 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.524+0000 7fe5c58f6640 1 -- start start 2024-03-25T18:12:12.525 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.524+0000 7fe5c58f6640 10 monclient: init 2024-03-25T18:12:12.525 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.524+0000 7fe5c58f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:12.526 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.524+0000 7fe5c58f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:12:12.526 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.524+0000 7fe5c58f6640 10 monclient(hunting): picked mon.noname-c con 0x7fe5c00a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:12:12.526 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.524+0000 7fe5c58f6640 10 monclient(hunting): picked mon.noname-a con 0x7fe5c013f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:12:12.526 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.524+0000 7fe5c58f6640 10 monclient(hunting): picked mon.noname-b con 0x7fe5c01430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:12:12.526 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.524+0000 7fe5c58f6640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5c0142520 con 0x7fe5c013f710 2024-03-25T18:12:12.526 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.524+0000 7fe5c58f6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5c00b28a0 con 0x7fe5c00a98f0 2024-03-25T18:12:12.526 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.524+0000 7fe5c58f6640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5c00b2a30 con 0x7fe5c01430c0 2024-03-25T18:12:12.526 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.524+0000 7fe5c58f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:12.526 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.524+0000 7fe5c58f6640 10 monclient(hunting): authenticate will time out at 7481.583924s 2024-03-25T18:12:12.544 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.544+0000 7fe5b7fff640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7fe5c013f710 0x7fe5c01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:41596/0 (socket says 172.21.15.66:41596) 2024-03-25T18:12:12.544 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.544+0000 7fe5b7fff640 1 -- 172.21.15.66:0/3890597967 learned_addr learned my addr 172.21.15.66:0/3890597967 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:12:12.554 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.553+0000 7fe5be7fc640 1 -- 172.21.15.66:0/3890597967 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036438665 0 0) 0x7fe5c00b28a0 con 0x7fe5c00a98f0 2024-03-25T18:12:12.554 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.553+0000 7fe5be7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:12.554 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.553+0000 7fe5be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:12.554 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.553+0000 7fe5be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:12.554 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.553+0000 7fe5be7fc640 10 monclient(hunting): my global_id is 4591 2024-03-25T18:12:12.554 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.553+0000 7fe5be7fc640 1 -- 172.21.15.66:0/3890597967 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5ac003520 con 0x7fe5c00a98f0 2024-03-25T18:12:12.571 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.570+0000 7fe5be7fc640 1 -- 172.21.15.66:0/3890597967 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2356151547 0 0) 0x7fe5ac003520 con 0x7fe5c00a98f0 2024-03-25T18:12:12.571 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.570+0000 7fe5be7fc640 1 -- 172.21.15.66:0/3890597967 >> v1:172.21.15.66:6790/0 conn(0x7fe5c01430c0 legacy=0x7fe5c014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:12.571 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.570+0000 7fe5be7fc640 1 -- 172.21.15.66:0/3890597967 >> v1:172.21.15.66:6789/0 conn(0x7fe5c013f710 legacy=0x7fe5c01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:12.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.570+0000 7fe5be7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:12:12.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.570+0000 7fe5be7fc640 1 monclient: found mon.noname-c 2024-03-25T18:12:12.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.570+0000 7fe5be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:12.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.570+0000 7fe5be7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:12:12.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.570+0000 7fe5be7fc640 1 -- 172.21.15.66:0/3890597967 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c00b44c0 con 0x7fe5c00a98f0 2024-03-25T18:12:12.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.570+0000 7fe5be7fc640 10 monclient: _finish_auth 0 2024-03-25T18:12:12.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.570+0000 7fe5be7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:12:12.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.570+0000 7fe5be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:12.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.570+0000 7fe5be7fc640 1 -- 172.21.15.66:0/3890597967 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fe5a8001690 con 0x7fe5c00a98f0 2024-03-25T18:12:12.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.570+0000 7fe5be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:12.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.570+0000 7fe5be7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:12.572 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.570+0000 7fe5be7fc640 10 monclient: dump: 2024-03-25T18:12:12.572 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:12.572 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:12.573 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:12.573 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:12.573 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:12.573 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:12.573 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:12.573 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:12.573 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:12.573 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:12.573 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.570+0000 7fe5c58f6640 5 monclient: authenticate success, global_id 4591 2024-03-25T18:12:12.573 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.570+0000 7fe5c58f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:12.602 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.601+0000 7fe5be7fc640 1 -- 172.21.15.66:0/3890597967 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5a80017f0 con 0x7fe5c00a98f0 2024-03-25T18:12:12.602 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.601+0000 7fe5be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:12.602 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.601+0000 7fe5be7fc640 1 -- 172.21.15.66:0/3890597967 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fe5a8001c00 con 0x7fe5c00a98f0 2024-03-25T18:12:12.602 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.601+0000 7fe5be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:12.602 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.601+0000 7fe5be7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:12.602 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.601+0000 7fe5be7fc640 10 monclient: dump: 2024-03-25T18:12:12.602 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:12.602 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:12.602 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:12.602 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:12.602 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:12.603 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:12.603 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:12.603 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:12.603 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:12.603 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:12.603 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.601+0000 7fe5c58f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:12.603 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.601+0000 7fe5c58f6640 4 set_mon_vals no callback set 2024-03-25T18:12:12.605 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.602+0000 7fe5c58f6640 10 monclient: shutdown 2024-03-25T18:12:12.605 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.602+0000 7fe5c58f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:12.605 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.602+0000 7fe5c58f6640 1 -- 172.21.15.66:0/3890597967 >> v1:172.21.15.193:6789/0 conn(0x7fe5c00a98f0 legacy=0x7fe5c00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:12.605 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.602+0000 7fe5c58f6640 1 -- 172.21.15.66:0/3890597967 shutdown_connections 2024-03-25T18:12:12.605 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.602+0000 7fe5c58f6640 1 -- 172.21.15.66:0/3890597967 >> 172.21.15.66:0/3890597967 conn(0x7fe5c00b7ea0 msgr2=0x7fe5c00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:12.605 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.602+0000 7fe5c58f6640 1 -- 172.21.15.66:0/3890597967 shutdown_connections 2024-03-25T18:12:12.606 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.602+0000 7fe5c58f6640 1 -- 172.21.15.66:0/3890597967 wait complete. 2024-03-25T18:12:12.606 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.603+0000 7fe5c58f6640 10 monclient: build_initial_monmap 2024-03-25T18:12:12.606 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.603+0000 7fe5c58f6640 10 monclient: monmap: 2024-03-25T18:12:12.606 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:12:12.606 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:12.606 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:12:12.606 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:12:12.606 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:12.606 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:12:12.606 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:12:12.606 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:12:12.606 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:12:12.607 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:12.607 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.603+0000 7fe5c58f6640 1 librados: starting msgr at 2024-03-25T18:12:12.607 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.603+0000 7fe5c58f6640 1 librados: starting objecter 2024-03-25T18:12:12.607 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.603+0000 7fe5c58f6640 1 Processor -- start 2024-03-25T18:12:12.607 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.603+0000 7fe5c58f6640 1 -- start start 2024-03-25T18:12:12.607 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.603+0000 7fe5c58f6640 1 librados: setting wanted keys 2024-03-25T18:12:12.607 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.603+0000 7fe5c58f6640 1 librados: calling monclient init 2024-03-25T18:12:12.607 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.603+0000 7fe5c58f6640 10 monclient: init 2024-03-25T18:12:12.607 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.604+0000 7fe5c58f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:12.607 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.604+0000 7fe5c58f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:12:12.607 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.604+0000 7fe5c58f6640 10 monclient(hunting): picked mon.noname-b con 0x7fe5c00a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:12:12.607 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.604+0000 7fe5c58f6640 10 monclient(hunting): picked mon.noname-a con 0x7fe5c013f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:12:12.607 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.604+0000 7fe5c58f6640 10 monclient(hunting): picked mon.noname-c con 0x7fe5c01430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:12:12.607 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.604+0000 7fe5c58f6640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5c01db7d0 con 0x7fe5c013f710 2024-03-25T18:12:12.608 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.604+0000 7fe5c58f6640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5c00b1420 con 0x7fe5c00a98f0 2024-03-25T18:12:12.608 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.604+0000 7fe5c58f6640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5c00b15b0 con 0x7fe5c01430c0 2024-03-25T18:12:12.608 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.604+0000 7fe5c58f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:12.608 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.604+0000 7fe5c58f6640 10 monclient(hunting): authenticate will time out at 7481.663806s 2024-03-25T18:12:12.609 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.609+0000 7fe5bf7fe640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7fe5c01430c0 0x7fe5c01d4350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:41716/0 (socket says 172.21.15.66:41716) 2024-03-25T18:12:12.609 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.609+0000 7fe5bf7fe640 1 -- 172.21.15.66:0/685288633 learned_addr learned my addr 172.21.15.66:0/685288633 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:12:12.610 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.610+0000 7fe5bcff9640 10 client.?.objecter ms_handle_connect 0x7fe5c01430c0 2024-03-25T18:12:12.610 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.610+0000 7fe5bcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:12.617 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.616+0000 7fe5bcff9640 1 -- 172.21.15.66:0/685288633 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3444424959 0 0) 0x7fe5c00b15b0 con 0x7fe5c01430c0 2024-03-25T18:12:12.617 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.616+0000 7fe5bcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:12.617 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.616+0000 7fe5bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:12.617 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.616+0000 7fe5bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:12.617 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.616+0000 7fe5bcff9640 10 monclient(hunting): my global_id is 4595 2024-03-25T18:12:12.617 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.617+0000 7fe5bcff9640 1 -- 172.21.15.66:0/685288633 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5a0004eb0 con 0x7fe5c01430c0 2024-03-25T18:12:12.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.622+0000 7fe5bcff9640 10 client.?.objecter ms_handle_connect 0x7fe5c00a98f0 2024-03-25T18:12:12.623 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.622+0000 7fe5bcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:12.625 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.624+0000 7fe5bcff9640 10 client.?.objecter ms_handle_connect 0x7fe5c013f710 2024-03-25T18:12:12.625 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.624+0000 7fe5bcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:12.628 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.627+0000 7fe5bcff9640 1 -- 172.21.15.66:0/685288633 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 681128706 0 0) 0x7fe5a0004eb0 con 0x7fe5c01430c0 2024-03-25T18:12:12.628 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.627+0000 7fe5bcff9640 1 -- 172.21.15.66:0/685288633 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5c00b15b0 con 0x7fe5c01430c0 2024-03-25T18:12:12.628 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.627+0000 7fe5bcff9640 1 -- 172.21.15.66:0/685288633 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fe5b8001c90 con 0x7fe5c01430c0 2024-03-25T18:12:12.628 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.627+0000 7fe5bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:12.628 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.627+0000 7fe5bcff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:12.628 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.627+0000 7fe5bcff9640 10 monclient(hunting): dump: 2024-03-25T18:12:12.628 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:12.628 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:12.628 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:12.628 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:12.628 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:12.628 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:12.628 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:12.629 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:12.629 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:12.629 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:12.629 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.627+0000 7fe5bcff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:12.636 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.635+0000 7fe5bcff9640 1 -- 172.21.15.66:0/685288633 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1301322850 0 0) 0x7fe5c00b1420 con 0x7fe5c00a98f0 2024-03-25T18:12:12.636 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.635+0000 7fe5bcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:12.636 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.635+0000 7fe5bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:12.636 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.635+0000 7fe5bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:12.636 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.635+0000 7fe5bcff9640 10 monclient(hunting): my global_id is 4594 2024-03-25T18:12:12.636 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.635+0000 7fe5bcff9640 1 -- 172.21.15.66:0/685288633 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5a0004eb0 con 0x7fe5c00a98f0 2024-03-25T18:12:12.643 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.641+0000 7fe5bcff9640 1 -- 172.21.15.66:0/685288633 <== mon.2 v1:172.21.15.66:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 438316718 0 0) 0x7fe5c00b15b0 con 0x7fe5c01430c0 2024-03-25T18:12:12.643 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.641+0000 7fe5bcff9640 1 -- 172.21.15.66:0/685288633 >> v1:172.21.15.193:6789/0 conn(0x7fe5c00a98f0 legacy=0x7fe5c01daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:12.643 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.641+0000 7fe5bcff9640 1 -- 172.21.15.66:0/685288633 >> v1:172.21.15.66:6789/0 conn(0x7fe5c013f710 legacy=0x7fe5c00b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:12.643 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.641+0000 7fe5bcff9640 10 monclient: _finish_hunting 0 2024-03-25T18:12:12.643 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.641+0000 7fe5bcff9640 1 monclient: found mon.c 2024-03-25T18:12:12.643 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.641+0000 7fe5bcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:12.643 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.641+0000 7fe5bcff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:12:12.643 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.641+0000 7fe5bcff9640 1 -- 172.21.15.66:0/685288633 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c00b1740 con 0x7fe5c01430c0 2024-03-25T18:12:12.643 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.641+0000 7fe5bcff9640 10 monclient: _finish_auth 0 2024-03-25T18:12:12.643 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.641+0000 7fe5bcff9640 10 monclient: _check_auth_tickets 2024-03-25T18:12:12.643 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.641+0000 7fe5bcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:12.643 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.641+0000 7fe5c58f6640 5 monclient: authenticate success, global_id 4595 2024-03-25T18:12:12.643 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.642+0000 7fe5c58f6640 10 monclient: _renew_subs 2024-03-25T18:12:12.643 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.642+0000 7fe5c58f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:12:12.643 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.642+0000 7fe5c58f6640 1 -- 172.21.15.66:0/685288633 --> v1:172.21.15.66:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5c01db1a0 con 0x7fe5c01430c0 2024-03-25T18:12:12.644 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.642+0000 7fe5c58f6640 10 client.4595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:12.644 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.642+0000 7fe5c58f6640 10 monclient: _renew_subs 2024-03-25T18:12:12.644 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.642+0000 7fe5c58f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:12:12.644 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.642+0000 7fe5c58f6640 1 -- 172.21.15.66:0/685288633 --> v1:172.21.15.66:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5c00b08f0 con 0x7fe5c01430c0 2024-03-25T18:12:12.644 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.642+0000 7fe5c58f6640 1 librados: init done 2024-03-25T18:12:12.644 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.643+0000 7fe5c58f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:12.644 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.643+0000 7fe5c58f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:12.645 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.643+0000 7fe5c58f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:12:12.645 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.643+0000 7fe5c58f6640 1 -- 172.21.15.66:0/685288633 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5c00627b0 con 0x7fe5c01430c0 2024-03-25T18:12:12.959 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.958+0000 7fe5bcff9640 1 -- 172.21.15.66:0/685288633 <== mon.2 v1:172.21.15.66:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5b8003990 con 0x7fe5c01430c0 2024-03-25T18:12:12.959 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.958+0000 7fe5bcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:12.959 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.958+0000 7fe5bcff9640 1 -- 172.21.15.66:0/685288633 <== mon.2 v1:172.21.15.66:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fe5b8003d20 con 0x7fe5c01430c0 2024-03-25T18:12:12.960 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.958+0000 7fe5bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:12.960 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.958+0000 7fe5bcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:12.960 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.958+0000 7fe5bcff9640 10 monclient: dump: 2024-03-25T18:12:12.960 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:12.960 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:12.960 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:12.960 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:12.960 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:12.960 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:12.960 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:12.960 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:12.960 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:12.961 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:12.961 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.958+0000 7fe5bdffb640 4 set_mon_vals no callback set 2024-03-25T18:12:12.961 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.959+0000 7fe5bcff9640 1 -- 172.21.15.66:0/685288633 <== mon.2 v1:172.21.15.66:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7fe5b8003f40 con 0x7fe5c01430c0 2024-03-25T18:12:12.961 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.960+0000 7fe5beffd640 10 monclient: get_auth_request con 0x7fe5a006dd90 auth_method 0 2024-03-25T18:12:12.961 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.960+0000 7fe5bcff9640 10 client.4595.objecter ms_handle_connect 0x7fe5a006dd90 2024-03-25T18:12:12.966 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.966+0000 7fe5bcff9640 1 -- 172.21.15.66:0/685288633 <== mon.2 v1:172.21.15.66:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7fe5b8054a80 con 0x7fe5c01430c0 2024-03-25T18:12:12.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.966+0000 7fe5bcff9640 10 client.4595.objecter ms_dispatch 0x7fe5c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:12.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.966+0000 7fe5bcff9640 3 client.4595.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:12.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.966+0000 7fe5bcff9640 3 client.4595.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:12.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.966+0000 7fe5bcff9640 20 client.4595.objecter dump_active .. 0 homeless 2024-03-25T18:12:12.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.966+0000 7fe5bcff9640 1 -- 172.21.15.66:0/685288633 <== mon.2 v1:172.21.15.66:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe5b8050590 con 0x7fe5c01430c0 2024-03-25T18:12:12.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.966+0000 7fe5bcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:12.967 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:12.966+0000 7fe5bcff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:13.223 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:13.222+0000 7fe5c58f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:13.223 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:13.222+0000 7fe5c58f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:13.223 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:13.222+0000 7fe5c58f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:12:13.223 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:13.222+0000 7fe5c58f6640 1 -- 172.21.15.66:0/685288633 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5c00abb10 con 0x7fe5c01430c0 2024-03-25T18:12:13.252 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:13.251+0000 7fe5bcff9640 1 -- 172.21.15.66:0/685288633 <== mon.2 v1:172.21.15.66:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7fe5b80524c0 con 0x7fe5c01430c0 2024-03-25T18:12:13.252 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:13.251+0000 7fe5bcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:13.252 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:13.251+0000 7fe5bcff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:13.252 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:12:13.252 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:13.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:13.253+0000 7fe5c58f6640 10 librados: watch_flush enter 2024-03-25T18:12:13.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:13.253+0000 7fe5c58f6640 10 librados: watch_flush exit 2024-03-25T18:12:13.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:13.253+0000 7fe5c58f6640 20 client.4595.objecter shutdown clearing up homeless session... 2024-03-25T18:12:13.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:13.253+0000 7fe5c58f6640 10 client.4595.objecter successfully canceled tick 2024-03-25T18:12:13.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:13.253+0000 7fe5c58f6640 1 -- 172.21.15.66:0/685288633 >> v1:172.21.15.193:6800/2275676803 conn(0x7fe5a006dd90 legacy=0x7fe5a0070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:13.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:13.253+0000 7fe5c58f6640 10 monclient: shutdown 2024-03-25T18:12:13.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:13.253+0000 7fe5c58f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:13.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:13.253+0000 7fe5c58f6640 1 -- 172.21.15.66:0/685288633 >> v1:172.21.15.66:6790/0 conn(0x7fe5c01430c0 legacy=0x7fe5c01d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:13.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:13.254+0000 7fe5c58f6640 1 -- 172.21.15.66:0/685288633 shutdown_connections 2024-03-25T18:12:13.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:13.254+0000 7fe5c58f6640 1 -- 172.21.15.66:0/685288633 >> 172.21.15.66:0/685288633 conn(0x7fe5c00b7ea0 msgr2=0x7fe5c013c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:13.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:13.254+0000 7fe5c58f6640 1 -- 172.21.15.66:0/685288633 shutdown_connections 2024-03-25T18:12:13.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:13.254+0000 7fe5c58f6640 1 -- 172.21.15.66:0/685288633 wait complete. 2024-03-25T18:12:13.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:13.254+0000 7fe5c58f6640 1 librados: shutdown 2024-03-25T18:12:13.267 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:20.073 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:20.192 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.191+0000 7faa0250e640 10 monclient: get_monmap_and_config 2024-03-25T18:12:20.192 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.191+0000 7faa0250e640 10 monclient: build_initial_monmap 2024-03-25T18:12:20.192 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.191+0000 7faa0250e640 10 monclient: monmap: 2024-03-25T18:12:20.192 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:12:20.192 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:20.193 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:12:20.192389+0000 2024-03-25T18:12:20.193 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:12:20.192389+0000 2024-03-25T18:12:20.193 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:20.193 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:20.193 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:12:20.193 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:12:20.193 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:12:20.193 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:20.194 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.192+0000 7faa0250e640 1 Processor -- start 2024-03-25T18:12:20.194 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.193+0000 7faa0250e640 1 -- start start 2024-03-25T18:12:20.194 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.193+0000 7faa0250e640 10 monclient: init 2024-03-25T18:12:20.194 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.193+0000 7faa0250e640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:20.194 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.193+0000 7faa0250e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:12:20.194 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.193+0000 7faa0250e640 10 monclient(hunting): picked mon.noname-b con 0x7fa9fc0a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:12:20.195 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.193+0000 7faa0250e640 10 monclient(hunting): picked mon.noname-c con 0x7fa9fc13f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:12:20.195 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.193+0000 7faa0250e640 10 monclient(hunting): picked mon.noname-a con 0x7fa9fc1430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:12:20.195 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.193+0000 7faa0250e640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9fc142520 con 0x7fa9fc1430c0 2024-03-25T18:12:20.195 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.193+0000 7faa0250e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9fc0b28a0 con 0x7fa9fc13f710 2024-03-25T18:12:20.195 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.193+0000 7faa0250e640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9fc0b2a30 con 0x7fa9fc0a98f0 2024-03-25T18:12:20.195 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.193+0000 7faa0250e640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:20.195 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.193+0000 7faa0250e640 10 monclient(hunting): authenticate will time out at 7489.252954s 2024-03-25T18:12:20.203 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.202+0000 7fa9fb7fe640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fa9fc13f710 0x7fa9fc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:51652/0 (socket says 172.21.15.66:51652) 2024-03-25T18:12:20.203 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.202+0000 7fa9fb7fe640 1 -- 172.21.15.66:0/3876478601 learned_addr learned my addr 172.21.15.66:0/3876478601 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:12:20.212 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.211+0000 7fa9faffd640 1 -- 172.21.15.66:0/3876478601 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1086827344 0 0) 0x7fa9fc142520 con 0x7fa9fc1430c0 2024-03-25T18:12:20.212 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.211+0000 7fa9faffd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:20.212 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.211+0000 7fa9faffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:20.212 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.211+0000 7fa9faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:20.212 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.211+0000 7fa9faffd640 10 monclient(hunting): my global_id is 4602 2024-03-25T18:12:20.212 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.211+0000 7fa9faffd640 1 -- 172.21.15.66:0/3876478601 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9e4003520 con 0x7fa9fc1430c0 2024-03-25T18:12:20.223 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.222+0000 7fa9faffd640 1 -- 172.21.15.66:0/3876478601 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006948552 0 0) 0x7fa9fc0b28a0 con 0x7fa9fc13f710 2024-03-25T18:12:20.223 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.222+0000 7fa9faffd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:20.223 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.222+0000 7fa9faffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:20.223 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.222+0000 7fa9faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:20.223 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.222+0000 7fa9faffd640 10 monclient(hunting): my global_id is 4597 2024-03-25T18:12:20.223 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.222+0000 7fa9faffd640 1 -- 172.21.15.66:0/3876478601 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9fc142520 con 0x7fa9fc13f710 2024-03-25T18:12:20.227 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.226+0000 7fa9faffd640 1 -- 172.21.15.66:0/3876478601 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 472367019 0 0) 0x7fa9fc0b2a30 con 0x7fa9fc0a98f0 2024-03-25T18:12:20.227 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.226+0000 7fa9faffd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:20.227 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.226+0000 7fa9faffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:20.227 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.226+0000 7fa9faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:20.227 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.226+0000 7fa9faffd640 10 monclient(hunting): my global_id is 4598 2024-03-25T18:12:20.227 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.226+0000 7fa9faffd640 1 -- 172.21.15.66:0/3876478601 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9fc0b28a0 con 0x7fa9fc0a98f0 2024-03-25T18:12:20.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.232+0000 7fa9faffd640 1 -- 172.21.15.66:0/3876478601 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2472880970 0 0) 0x7fa9e4003520 con 0x7fa9fc1430c0 2024-03-25T18:12:20.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.232+0000 7fa9faffd640 1 -- 172.21.15.66:0/3876478601 >> v1:172.21.15.66:6790/0 conn(0x7fa9fc0a98f0 legacy=0x7fa9fc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:20.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.232+0000 7fa9faffd640 1 -- 172.21.15.66:0/3876478601 >> v1:172.21.15.193:6789/0 conn(0x7fa9fc13f710 legacy=0x7fa9fc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:20.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.232+0000 7fa9faffd640 10 monclient: _finish_hunting 0 2024-03-25T18:12:20.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.232+0000 7fa9faffd640 1 monclient: found mon.noname-a 2024-03-25T18:12:20.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.232+0000 7fa9faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:20.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.232+0000 7fa9faffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.66:6789/0 2024-03-25T18:12:20.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.232+0000 7fa9faffd640 1 -- 172.21.15.66:0/3876478601 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9fc0b44c0 con 0x7fa9fc1430c0 2024-03-25T18:12:20.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.232+0000 7fa9faffd640 10 monclient: _finish_auth 0 2024-03-25T18:12:20.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.232+0000 7fa9faffd640 10 monclient: _check_auth_tickets 2024-03-25T18:12:20.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.232+0000 7fa9faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:20.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.232+0000 7fa9faffd640 1 -- 172.21.15.66:0/3876478601 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fa9f4001a10 con 0x7fa9fc1430c0 2024-03-25T18:12:20.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.232+0000 7faa0250e640 5 monclient: authenticate success, global_id 4602 2024-03-25T18:12:20.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.232+0000 7faa0250e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:20.234 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.232+0000 7fa9faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:20.234 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.232+0000 7fa9faffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:12:20.234 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.232+0000 7fa9faffd640 10 monclient: dump: 2024-03-25T18:12:20.234 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:20.234 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:20.234 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:20.234 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:20.234 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:20.234 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:20.234 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:20.234 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:20.234 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:20.234 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:20.234 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.232+0000 7faa0250e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:20.250 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.250+0000 7fa9faffd640 1 -- 172.21.15.66:0/3876478601 <== mon.0 v1:172.21.15.66:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9f4001b70 con 0x7fa9fc1430c0 2024-03-25T18:12:20.250 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.250+0000 7fa9faffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:20.250 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.250+0000 7fa9faffd640 1 -- 172.21.15.66:0/3876478601 <== mon.0 v1:172.21.15.66:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fa9f4001f80 con 0x7fa9fc1430c0 2024-03-25T18:12:20.250 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.250+0000 7fa9faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:20.250 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.250+0000 7fa9faffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:20.250 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.250+0000 7fa9faffd640 10 monclient: dump: 2024-03-25T18:12:20.251 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:20.251 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:20.251 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:20.251 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:20.251 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:20.251 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:20.251 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:20.251 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:20.251 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:20.251 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:20.251 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.250+0000 7faa0250e640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:20.251 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.250+0000 7faa0250e640 4 set_mon_vals no callback set 2024-03-25T18:12:20.252 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.251+0000 7faa0250e640 10 monclient: shutdown 2024-03-25T18:12:20.252 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.251+0000 7faa0250e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:20.252 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.251+0000 7faa0250e640 1 -- 172.21.15.66:0/3876478601 >> v1:172.21.15.66:6789/0 conn(0x7fa9fc1430c0 legacy=0x7fa9fc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:20.252 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.251+0000 7faa0250e640 1 -- 172.21.15.66:0/3876478601 shutdown_connections 2024-03-25T18:12:20.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.251+0000 7faa0250e640 1 -- 172.21.15.66:0/3876478601 >> 172.21.15.66:0/3876478601 conn(0x7fa9fc0b7ea0 msgr2=0x7fa9fc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:20.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.251+0000 7faa0250e640 1 -- 172.21.15.66:0/3876478601 shutdown_connections 2024-03-25T18:12:20.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.251+0000 7faa0250e640 1 -- 172.21.15.66:0/3876478601 wait complete. 2024-03-25T18:12:20.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.252+0000 7faa0250e640 10 monclient: build_initial_monmap 2024-03-25T18:12:20.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.252+0000 7faa0250e640 10 monclient: monmap: 2024-03-25T18:12:20.253 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:12:20.253 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:20.253 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:12:20.253 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:12:20.253 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:20.253 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:12:20.253 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:12:20.253 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:12:20.253 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:12:20.253 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:20.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.252+0000 7faa0250e640 1 librados: starting msgr at 2024-03-25T18:12:20.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.252+0000 7faa0250e640 1 librados: starting objecter 2024-03-25T18:12:20.254 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.252+0000 7faa0250e640 1 Processor -- start 2024-03-25T18:12:20.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.252+0000 7faa0250e640 1 -- start start 2024-03-25T18:12:20.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.252+0000 7faa0250e640 1 librados: setting wanted keys 2024-03-25T18:12:20.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.252+0000 7faa0250e640 1 librados: calling monclient init 2024-03-25T18:12:20.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.252+0000 7faa0250e640 10 monclient: init 2024-03-25T18:12:20.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.253+0000 7faa0250e640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:20.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.253+0000 7faa0250e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:12:20.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.253+0000 7faa0250e640 10 monclient(hunting): picked mon.noname-b con 0x7fa9fc0a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:12:20.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.253+0000 7faa0250e640 10 monclient(hunting): picked mon.noname-a con 0x7fa9fc13f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:12:20.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.253+0000 7faa0250e640 10 monclient(hunting): picked mon.noname-c con 0x7fa9fc1430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:12:20.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.253+0000 7faa0250e640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9fc136e20 con 0x7fa9fc13f710 2024-03-25T18:12:20.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.253+0000 7faa0250e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9fc0b7940 con 0x7fa9fc0a98f0 2024-03-25T18:12:20.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.253+0000 7faa0250e640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9fc1351a0 con 0x7fa9fc1430c0 2024-03-25T18:12:20.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.253+0000 7faa0250e640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:20.255 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.253+0000 7faa0250e640 10 monclient(hunting): authenticate will time out at 7489.312597s 2024-03-25T18:12:20.260 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.259+0000 7faa00aab640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7fa9fc1430c0 0x7fa9fc1e1fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:47488/0 (socket says 172.21.15.66:47488) 2024-03-25T18:12:20.260 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.259+0000 7faa00aab640 1 -- 172.21.15.66:0/1051361041 learned_addr learned my addr 172.21.15.66:0/1051361041 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:12:20.262 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.262+0000 7fa9f8ff9640 10 client.?.objecter ms_handle_connect 0x7fa9fc1430c0 2024-03-25T18:12:20.262 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.262+0000 7fa9f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:20.268 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.268+0000 7fa9f8ff9640 10 client.?.objecter ms_handle_connect 0x7fa9fc0a98f0 2024-03-25T18:12:20.268 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.268+0000 7fa9f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:20.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.273+0000 7fa9f8ff9640 10 client.?.objecter ms_handle_connect 0x7fa9fc13f710 2024-03-25T18:12:20.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.273+0000 7fa9f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:20.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.274+0000 7fa9f8ff9640 1 -- 172.21.15.66:0/1051361041 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2801175183 0 0) 0x7fa9fc0b7940 con 0x7fa9fc0a98f0 2024-03-25T18:12:20.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.274+0000 7fa9f8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:20.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.274+0000 7fa9f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:20.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.274+0000 7fa9f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:20.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.274+0000 7fa9f8ff9640 10 monclient(hunting): my global_id is 4600 2024-03-25T18:12:20.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.274+0000 7fa9f8ff9640 1 -- 172.21.15.66:0/1051361041 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9e8004eb0 con 0x7fa9fc0a98f0 2024-03-25T18:12:20.277 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.277+0000 7fa9f8ff9640 1 -- 172.21.15.66:0/1051361041 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2823335214 0 0) 0x7fa9fc1351a0 con 0x7fa9fc1430c0 2024-03-25T18:12:20.277 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.277+0000 7fa9f8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:20.277 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.277+0000 7fa9f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:20.278 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.277+0000 7fa9f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:20.278 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.277+0000 7fa9f8ff9640 10 monclient(hunting): my global_id is 4601 2024-03-25T18:12:20.278 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.277+0000 7fa9f8ff9640 1 -- 172.21.15.66:0/1051361041 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9fc0b7940 con 0x7fa9fc1430c0 2024-03-25T18:12:20.285 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.285+0000 7fa9f8ff9640 1 -- 172.21.15.66:0/1051361041 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3762279695 0 0) 0x7fa9fc0b7940 con 0x7fa9fc1430c0 2024-03-25T18:12:20.285 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.285+0000 7fa9f8ff9640 1 -- 172.21.15.66:0/1051361041 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9fc1351a0 con 0x7fa9fc1430c0 2024-03-25T18:12:20.285 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.285+0000 7fa9f8ff9640 1 -- 172.21.15.66:0/1051361041 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fa9f4001e80 con 0x7fa9fc1430c0 2024-03-25T18:12:20.285 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.285+0000 7fa9f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:20.285 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.285+0000 7fa9f8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:20.285 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.285+0000 7fa9f8ff9640 10 monclient(hunting): dump: 2024-03-25T18:12:20.286 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:20.286 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:20.286 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:20.286 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:20.286 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:20.286 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:20.286 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:20.286 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:20.286 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:20.286 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:20.286 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.285+0000 7fa9f8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:20.291 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.290+0000 7fa9f8ff9640 1 -- 172.21.15.66:0/1051361041 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3677961323 0 0) 0x7fa9e8004eb0 con 0x7fa9fc0a98f0 2024-03-25T18:12:20.291 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.290+0000 7fa9f8ff9640 1 -- 172.21.15.66:0/1051361041 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9fc0b7940 con 0x7fa9fc0a98f0 2024-03-25T18:12:20.291 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.290+0000 7fa9f8ff9640 1 -- 172.21.15.66:0/1051361041 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fa9ec0018f0 con 0x7fa9fc0a98f0 2024-03-25T18:12:20.291 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.290+0000 7fa9f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:20.291 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.290+0000 7fa9f8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:20.291 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.290+0000 7fa9f8ff9640 10 monclient(hunting): dump: 2024-03-25T18:12:20.291 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:20.291 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:20.291 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:20.291 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:20.291 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:20.291 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:20.291 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:20.291 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:20.291 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:20.292 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:20.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.295+0000 7fa9f8ff9640 1 -- 172.21.15.66:0/1051361041 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1810096183 0 0) 0x7fa9fc136e20 con 0x7fa9fc13f710 2024-03-25T18:12:20.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.295+0000 7fa9f8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:20.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.295+0000 7fa9f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:20.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.295+0000 7fa9f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:20.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.295+0000 7fa9f8ff9640 10 monclient(hunting): my global_id is 4605 2024-03-25T18:12:20.296 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.295+0000 7fa9f8ff9640 1 -- 172.21.15.66:0/1051361041 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9e8004eb0 con 0x7fa9fc13f710 2024-03-25T18:12:20.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.296+0000 7fa9f8ff9640 1 -- 172.21.15.66:0/1051361041 <== mon.2 v1:172.21.15.66:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2667934722 0 0) 0x7fa9fc1351a0 con 0x7fa9fc1430c0 2024-03-25T18:12:20.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.296+0000 7fa9f8ff9640 1 -- 172.21.15.66:0/1051361041 >> v1:172.21.15.193:6789/0 conn(0x7fa9fc0a98f0 legacy=0x7fa9fc1daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:20.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.296+0000 7fa9f8ff9640 1 -- 172.21.15.66:0/1051361041 >> v1:172.21.15.66:6789/0 conn(0x7fa9fc13f710 legacy=0x7fa9fc1db1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:20.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.296+0000 7fa9f8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:12:20.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.296+0000 7fa9f8ff9640 1 monclient: found mon.c 2024-03-25T18:12:20.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.296+0000 7fa9f8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:20.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.296+0000 7fa9f8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:12:20.297 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.296+0000 7fa9f8ff9640 1 -- 172.21.15.66:0/1051361041 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9fc135330 con 0x7fa9fc1430c0 2024-03-25T18:12:20.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.296+0000 7fa9f8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:12:20.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.296+0000 7fa9f8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:12:20.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.296+0000 7fa9f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:20.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.296+0000 7faa0250e640 5 monclient: authenticate success, global_id 4601 2024-03-25T18:12:20.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.296+0000 7faa0250e640 10 monclient: _renew_subs 2024-03-25T18:12:20.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.296+0000 7faa0250e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:12:20.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.296+0000 7faa0250e640 1 -- 172.21.15.66:0/1051361041 --> v1:172.21.15.66:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9fc135560 con 0x7fa9fc1430c0 2024-03-25T18:12:20.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.296+0000 7faa0250e640 10 client.4601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:20.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.296+0000 7faa0250e640 10 monclient: _renew_subs 2024-03-25T18:12:20.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.296+0000 7faa0250e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:12:20.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.296+0000 7faa0250e640 1 -- 172.21.15.66:0/1051361041 --> v1:172.21.15.66:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9fc13b490 con 0x7fa9fc1430c0 2024-03-25T18:12:20.298 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.296+0000 7faa0250e640 1 librados: init done 2024-03-25T18:12:20.299 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.297+0000 7faa0250e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:20.299 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.297+0000 7faa0250e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:20.299 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.297+0000 7faa0250e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:12:20.299 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.297+0000 7faa0250e640 1 -- 172.21.15.66:0/1051361041 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9fc0b7ad0 con 0x7fa9fc1430c0 2024-03-25T18:12:20.522 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.522+0000 7fa9f8ff9640 1 -- 172.21.15.66:0/1051361041 <== mon.2 v1:172.21.15.66:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9f4001590 con 0x7fa9fc1430c0 2024-03-25T18:12:20.522 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.522+0000 7fa9f8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:20.523 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.522+0000 7fa9f8ff9640 1 -- 172.21.15.66:0/1051361041 <== mon.2 v1:172.21.15.66:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fa9f4003c20 con 0x7fa9fc1430c0 2024-03-25T18:12:20.523 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.522+0000 7fa9f8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:20.523 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.522+0000 7fa9f8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:20.523 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.522+0000 7fa9f8ff9640 10 monclient: dump: 2024-03-25T18:12:20.523 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:20.523 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:20.523 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:20.523 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:20.523 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:20.523 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:20.523 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:20.523 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:20.523 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:20.523 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:20.523 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.522+0000 7fa9fa7fc640 4 set_mon_vals no callback set 2024-03-25T18:12:20.614 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.613+0000 7fa9f8ff9640 1 -- 172.21.15.66:0/1051361041 <== mon.2 v1:172.21.15.66:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7fa9f4003d80 con 0x7fa9fc1430c0 2024-03-25T18:12:20.614 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.614+0000 7fa9fbfff640 10 monclient: get_auth_request con 0x7fa9e806df50 auth_method 0 2024-03-25T18:12:20.614 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.614+0000 7fa9f8ff9640 10 client.4601.objecter ms_handle_connect 0x7fa9e806df50 2024-03-25T18:12:20.619 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.619+0000 7fa9f8ff9640 1 -- 172.21.15.66:0/1051361041 <== mon.2 v1:172.21.15.66:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7fa9f4054a10 con 0x7fa9fc1430c0 2024-03-25T18:12:20.620 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.619+0000 7fa9f8ff9640 10 client.4601.objecter ms_dispatch 0x7fa9fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:20.620 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.619+0000 7fa9f8ff9640 3 client.4601.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:20.620 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.619+0000 7fa9f8ff9640 3 client.4601.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:20.620 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.619+0000 7fa9f8ff9640 20 client.4601.objecter dump_active .. 0 homeless 2024-03-25T18:12:20.620 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.619+0000 7fa9f8ff9640 1 -- 172.21.15.66:0/1051361041 <== mon.2 v1:172.21.15.66:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa9f4050350 con 0x7fa9fc1430c0 2024-03-25T18:12:20.620 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.619+0000 7fa9f8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:20.620 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.619+0000 7fa9f8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:20.850 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.849+0000 7faa0250e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:20.850 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.849+0000 7faa0250e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:20.850 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.849+0000 7faa0250e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:12:20.850 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.849+0000 7faa0250e640 1 -- 172.21.15.66:0/1051361041 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9fc0b7730 con 0x7fa9fc1430c0 2024-03-25T18:12:20.873 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.873+0000 7fa9f8ff9640 1 -- 172.21.15.66:0/1051361041 <== mon.2 v1:172.21.15.66:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7fa9f4052450 con 0x7fa9fc1430c0 2024-03-25T18:12:20.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.873+0000 7fa9f8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:20.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.873+0000 7fa9f8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:20.874 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:12:20.874 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:20.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.875+0000 7faa0250e640 10 librados: watch_flush enter 2024-03-25T18:12:20.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.875+0000 7faa0250e640 10 librados: watch_flush exit 2024-03-25T18:12:20.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.875+0000 7faa0250e640 20 client.4601.objecter shutdown clearing up homeless session... 2024-03-25T18:12:20.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.875+0000 7faa0250e640 10 client.4601.objecter successfully canceled tick 2024-03-25T18:12:20.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.875+0000 7faa0250e640 1 -- 172.21.15.66:0/1051361041 >> v1:172.21.15.193:6800/2275676803 conn(0x7fa9e806df50 legacy=0x7fa9e8070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:20.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.875+0000 7faa0250e640 10 monclient: shutdown 2024-03-25T18:12:20.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.875+0000 7faa0250e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:20.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.875+0000 7faa0250e640 1 -- 172.21.15.66:0/1051361041 >> v1:172.21.15.66:6790/0 conn(0x7fa9fc1430c0 legacy=0x7fa9fc1e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:20.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.875+0000 7faa0250e640 1 -- 172.21.15.66:0/1051361041 shutdown_connections 2024-03-25T18:12:20.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.875+0000 7faa0250e640 1 -- 172.21.15.66:0/1051361041 >> 172.21.15.66:0/1051361041 conn(0x7fa9fc0b7ea0 msgr2=0x7fa9fc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:20.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.876+0000 7faa0250e640 1 -- 172.21.15.66:0/1051361041 shutdown_connections 2024-03-25T18:12:20.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.876+0000 7faa0250e640 1 -- 172.21.15.66:0/1051361041 wait complete. 2024-03-25T18:12:20.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:20.876+0000 7faa0250e640 1 librados: shutdown 2024-03-25T18:12:20.887 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:27.694 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:27.814 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.814+0000 7f994c97b640 10 monclient: get_monmap_and_config 2024-03-25T18:12:27.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.814+0000 7f994c97b640 10 monclient: build_initial_monmap 2024-03-25T18:12:27.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.814+0000 7f994c97b640 10 monclient: monmap: 2024-03-25T18:12:27.815 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:12:27.815 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:27.815 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:12:27.815011+0000 2024-03-25T18:12:27.815 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:12:27.815011+0000 2024-03-25T18:12:27.815 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:27.815 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:27.815 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:12:27.815 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:12:27.815 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:12:27.815 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:27.816 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.815+0000 7f994c97b640 1 Processor -- start 2024-03-25T18:12:27.816 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.815+0000 7f994c97b640 1 -- start start 2024-03-25T18:12:27.816 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.815+0000 7f994c97b640 10 monclient: init 2024-03-25T18:12:27.817 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.816+0000 7f994c97b640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:27.817 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.816+0000 7f994c97b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:12:27.817 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.816+0000 7f994c97b640 10 monclient(hunting): picked mon.noname-b con 0x7f99480a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:12:27.817 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.816+0000 7f994c97b640 10 monclient(hunting): picked mon.noname-a con 0x7f994813f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:12:27.817 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.816+0000 7f994c97b640 10 monclient(hunting): picked mon.noname-c con 0x7f99481430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:12:27.817 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.816+0000 7f994c97b640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9948142520 con 0x7f994813f710 2024-03-25T18:12:27.817 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.816+0000 7f994c97b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99480b28a0 con 0x7f99481430c0 2024-03-25T18:12:27.817 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.816+0000 7f994c97b640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99480b2a30 con 0x7f99480a98f0 2024-03-25T18:12:27.817 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.816+0000 7f994c97b640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:27.817 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.816+0000 7f994c97b640 10 monclient(hunting): authenticate will time out at 7496.875691s 2024-03-25T18:12:27.824 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.824+0000 7f9945d9b640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f994813f710 0x7f99481429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:37658/0 (socket says 172.21.15.66:37658) 2024-03-25T18:12:27.824 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.824+0000 7f9945d9b640 1 -- 172.21.15.66:0/90242999 learned_addr learned my addr 172.21.15.66:0/90242999 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:12:27.832 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.832+0000 7f994559a640 1 -- 172.21.15.66:0/90242999 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4112799348 0 0) 0x7f99480b28a0 con 0x7f99481430c0 2024-03-25T18:12:27.832 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.832+0000 7f994559a640 10 monclient(hunting): _negotiate 2024-03-25T18:12:27.833 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.832+0000 7f994559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:27.833 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.832+0000 7f994559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:27.833 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.832+0000 7f994559a640 10 monclient(hunting): my global_id is 4603 2024-03-25T18:12:27.833 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.832+0000 7f994559a640 1 -- 172.21.15.66:0/90242999 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9930003520 con 0x7f99481430c0 2024-03-25T18:12:27.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.839+0000 7f994559a640 1 -- 172.21.15.66:0/90242999 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2650575019 0 0) 0x7f9930003520 con 0x7f99481430c0 2024-03-25T18:12:27.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.839+0000 7f994559a640 1 -- 172.21.15.66:0/90242999 >> v1:172.21.15.66:6790/0 conn(0x7f99480a98f0 legacy=0x7f99480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:27.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.839+0000 7f994559a640 1 -- 172.21.15.66:0/90242999 >> v1:172.21.15.66:6789/0 conn(0x7f994813f710 legacy=0x7f99481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:27.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.839+0000 7f994559a640 10 monclient: _finish_hunting 0 2024-03-25T18:12:27.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.839+0000 7f994559a640 1 monclient: found mon.noname-c 2024-03-25T18:12:27.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.839+0000 7f994559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:27.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.839+0000 7f994559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:12:27.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.839+0000 7f994559a640 1 -- 172.21.15.66:0/90242999 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99480b44c0 con 0x7f99481430c0 2024-03-25T18:12:27.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.839+0000 7f994559a640 10 monclient: _finish_auth 0 2024-03-25T18:12:27.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.839+0000 7f994559a640 10 monclient: _check_auth_tickets 2024-03-25T18:12:27.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.839+0000 7f994559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:27.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.839+0000 7f994559a640 1 -- 172.21.15.66:0/90242999 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f9940001a10 con 0x7f99481430c0 2024-03-25T18:12:27.840 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.839+0000 7f994559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:27.841 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.839+0000 7f994559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:27.841 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.839+0000 7f994559a640 10 monclient: dump: 2024-03-25T18:12:27.841 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:27.841 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:27.841 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:27.841 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:27.841 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:27.841 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:27.841 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:27.841 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:27.841 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:27.841 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:27.841 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.839+0000 7f994c97b640 5 monclient: authenticate success, global_id 4603 2024-03-25T18:12:27.841 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.839+0000 7f994c97b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:27.864 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.864+0000 7f994559a640 1 -- 172.21.15.66:0/90242999 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9940001b70 con 0x7f99481430c0 2024-03-25T18:12:27.865 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.864+0000 7f994559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:27.865 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.864+0000 7f994559a640 1 -- 172.21.15.66:0/90242999 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f9940001f80 con 0x7f99481430c0 2024-03-25T18:12:27.865 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.864+0000 7f994559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:27.865 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.864+0000 7f994559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:27.865 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.864+0000 7f994559a640 10 monclient: dump: 2024-03-25T18:12:27.865 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:27.865 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:27.865 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:27.865 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:27.865 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:27.865 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:27.865 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:27.865 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:27.866 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:27.866 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:27.866 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.864+0000 7f994c97b640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:27.866 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.864+0000 7f994c97b640 4 set_mon_vals no callback set 2024-03-25T18:12:27.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.865+0000 7f994c97b640 10 monclient: shutdown 2024-03-25T18:12:27.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.865+0000 7f994c97b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:27.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.865+0000 7f994c97b640 1 -- 172.21.15.66:0/90242999 >> v1:172.21.15.193:6789/0 conn(0x7f99481430c0 legacy=0x7f994814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:27.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.865+0000 7f994c97b640 1 -- 172.21.15.66:0/90242999 shutdown_connections 2024-03-25T18:12:27.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.865+0000 7f994c97b640 1 -- 172.21.15.66:0/90242999 >> 172.21.15.66:0/90242999 conn(0x7f99480b7ea0 msgr2=0x7f99480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:27.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.865+0000 7f994c97b640 1 -- 172.21.15.66:0/90242999 shutdown_connections 2024-03-25T18:12:27.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.865+0000 7f994c97b640 1 -- 172.21.15.66:0/90242999 wait complete. 2024-03-25T18:12:27.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.866+0000 7f994c97b640 10 monclient: build_initial_monmap 2024-03-25T18:12:27.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.866+0000 7f994c97b640 10 monclient: monmap: 2024-03-25T18:12:27.867 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:12:27.867 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:27.868 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:12:27.868 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:12:27.868 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:27.868 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:12:27.868 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:12:27.868 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:12:27.868 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:12:27.868 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:27.868 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.867+0000 7f994c97b640 1 librados: starting msgr at 2024-03-25T18:12:27.868 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.867+0000 7f994c97b640 1 librados: starting objecter 2024-03-25T18:12:27.869 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.867+0000 7f994c97b640 1 Processor -- start 2024-03-25T18:12:27.870 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.867+0000 7f994c97b640 1 -- start start 2024-03-25T18:12:27.870 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.867+0000 7f994c97b640 1 librados: setting wanted keys 2024-03-25T18:12:27.870 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.867+0000 7f994c97b640 1 librados: calling monclient init 2024-03-25T18:12:27.870 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.867+0000 7f994c97b640 10 monclient: init 2024-03-25T18:12:27.870 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.868+0000 7f994c97b640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:27.870 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.868+0000 7f994c97b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:12:27.870 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.868+0000 7f994c97b640 10 monclient(hunting): picked mon.noname-c con 0x7f99480a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:12:27.870 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.868+0000 7f994c97b640 10 monclient(hunting): picked mon.noname-a con 0x7f994813f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:12:27.870 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.868+0000 7f994c97b640 10 monclient(hunting): picked mon.noname-b con 0x7f99481430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:12:27.870 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.868+0000 7f994c97b640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9948137ab0 con 0x7f994813f710 2024-03-25T18:12:27.870 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.868+0000 7f994c97b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9948138bd0 con 0x7f99481430c0 2024-03-25T18:12:27.870 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.868+0000 7f994c97b640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9948138e40 con 0x7f99480a98f0 2024-03-25T18:12:27.870 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.868+0000 7f994c97b640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:27.870 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.868+0000 7f994c97b640 10 monclient(hunting): authenticate will time out at 7496.927491s 2024-03-25T18:12:27.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.882+0000 7f9945d9b640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f994813f710 0x7f99481df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:37692/0 (socket says 172.21.15.66:37692) 2024-03-25T18:12:27.882 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.882+0000 7f9945d9b640 1 -- 172.21.15.66:0/3392241929 learned_addr learned my addr 172.21.15.66:0/3392241929 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:12:27.884 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.884+0000 7f9936ffd640 10 client.?.objecter ms_handle_connect 0x7f994813f710 2024-03-25T18:12:27.884 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.884+0000 7f9936ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:27.887 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.886+0000 7f9936ffd640 10 client.?.objecter ms_handle_connect 0x7f99480a98f0 2024-03-25T18:12:27.887 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.886+0000 7f9936ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:27.890 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.890+0000 7f9936ffd640 10 client.?.objecter ms_handle_connect 0x7f99481430c0 2024-03-25T18:12:27.891 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.890+0000 7f9936ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:27.896 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.895+0000 7f9936ffd640 1 -- 172.21.15.66:0/3392241929 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3089041783 0 0) 0x7f9948137ab0 con 0x7f994813f710 2024-03-25T18:12:27.896 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.895+0000 7f9936ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:27.896 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.895+0000 7f9936ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:27.896 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.895+0000 7f9936ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:27.896 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.895+0000 7f9936ffd640 10 monclient(hunting): my global_id is 4611 2024-03-25T18:12:27.896 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.895+0000 7f9936ffd640 1 -- 172.21.15.66:0/3392241929 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9928004eb0 con 0x7f994813f710 2024-03-25T18:12:27.907 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.907+0000 7f9936ffd640 1 -- 172.21.15.66:0/3392241929 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 733363577 0 0) 0x7f9948138bd0 con 0x7f99481430c0 2024-03-25T18:12:27.907 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.907+0000 7f9936ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:27.907 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.907+0000 7f9936ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:27.907 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.907+0000 7f9936ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:27.907 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.907+0000 7f9936ffd640 10 monclient(hunting): my global_id is 4606 2024-03-25T18:12:27.907 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.907+0000 7f9936ffd640 1 -- 172.21.15.66:0/3392241929 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9948137ab0 con 0x7f99481430c0 2024-03-25T18:12:27.915 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.915+0000 7f9936ffd640 1 -- 172.21.15.66:0/3392241929 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 397900629 0 0) 0x7f9928004eb0 con 0x7f994813f710 2024-03-25T18:12:27.915 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.915+0000 7f9936ffd640 1 -- 172.21.15.66:0/3392241929 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9948138bd0 con 0x7f994813f710 2024-03-25T18:12:27.916 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.915+0000 7f9936ffd640 1 -- 172.21.15.66:0/3392241929 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f993c0018b0 con 0x7f994813f710 2024-03-25T18:12:27.916 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.915+0000 7f9936ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:27.916 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.915+0000 7f9936ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:12:27.916 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.915+0000 7f9936ffd640 10 monclient(hunting): dump: 2024-03-25T18:12:27.916 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:27.916 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:27.916 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:27.916 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:27.916 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:27.916 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:27.916 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:27.916 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:27.916 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:27.916 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:27.916 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.915+0000 7f9936ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:27.917 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.915+0000 7f9936ffd640 1 -- 172.21.15.66:0/3392241929 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1572566444 0 0) 0x7f9948138e40 con 0x7f99480a98f0 2024-03-25T18:12:27.917 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.915+0000 7f9936ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:27.917 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.915+0000 7f9936ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:27.917 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.915+0000 7f9936ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:27.917 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.916+0000 7f9936ffd640 10 monclient(hunting): my global_id is 4607 2024-03-25T18:12:27.917 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.916+0000 7f9936ffd640 1 -- 172.21.15.66:0/3392241929 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9928004eb0 con 0x7f99480a98f0 2024-03-25T18:12:27.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.918+0000 7f9936ffd640 1 -- 172.21.15.66:0/3392241929 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 184544902 0 0) 0x7f9948137ab0 con 0x7f99481430c0 2024-03-25T18:12:27.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.918+0000 7f9936ffd640 1 -- 172.21.15.66:0/3392241929 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9948138e40 con 0x7f99481430c0 2024-03-25T18:12:27.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.918+0000 7f9936ffd640 1 -- 172.21.15.66:0/3392241929 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f9940001e80 con 0x7f99481430c0 2024-03-25T18:12:27.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.918+0000 7f9936ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:27.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.918+0000 7f9936ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:27.918 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.918+0000 7f9936ffd640 10 monclient(hunting): dump: 2024-03-25T18:12:27.919 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:27.919 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:27.919 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:27.919 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:27.919 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:27.919 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:27.919 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:27.919 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:27.919 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:27.919 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:27.930 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.929+0000 7f9936ffd640 1 -- 172.21.15.66:0/3392241929 <== mon.0 v1:172.21.15.66:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1297964764 0 0) 0x7f9948138bd0 con 0x7f994813f710 2024-03-25T18:12:27.931 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.929+0000 7f9936ffd640 1 -- 172.21.15.66:0/3392241929 >> v1:172.21.15.66:6790/0 conn(0x7f99480a98f0 legacy=0x7f9948137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:27.931 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.929+0000 7f9936ffd640 1 -- 172.21.15.66:0/3392241929 >> v1:172.21.15.193:6789/0 conn(0x7f99481430c0 legacy=0x7f99481e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:27.931 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.929+0000 7f9936ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:12:27.931 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.929+0000 7f9936ffd640 1 monclient: found mon.a 2024-03-25T18:12:27.931 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.929+0000 7f9936ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:27.931 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.929+0000 7f9936ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:12:27.931 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.929+0000 7f9936ffd640 1 -- 172.21.15.66:0/3392241929 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9948139120 con 0x7f994813f710 2024-03-25T18:12:27.931 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.929+0000 7f9936ffd640 10 monclient: _finish_auth 0 2024-03-25T18:12:27.931 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.929+0000 7f9936ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:12:27.931 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.929+0000 7f9936ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:27.931 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.929+0000 7f994c97b640 5 monclient: authenticate success, global_id 4611 2024-03-25T18:12:27.931 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.929+0000 7f994c97b640 10 monclient: _renew_subs 2024-03-25T18:12:27.931 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.929+0000 7f994c97b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:12:27.931 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.929+0000 7f994c97b640 1 -- 172.21.15.66:0/3392241929 --> v1:172.21.15.66:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9948139400 con 0x7f994813f710 2024-03-25T18:12:27.931 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.929+0000 7f994c97b640 10 client.4611.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:27.932 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.929+0000 7f994c97b640 10 monclient: _renew_subs 2024-03-25T18:12:27.932 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.929+0000 7f994c97b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:12:27.932 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.929+0000 7f994c97b640 1 -- 172.21.15.66:0/3392241929 --> v1:172.21.15.66:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f99480b7880 con 0x7f994813f710 2024-03-25T18:12:27.932 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.929+0000 7f994c97b640 1 librados: init done 2024-03-25T18:12:27.932 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.931+0000 7f994c97b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:27.932 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.931+0000 7f994c97b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:27.932 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.931+0000 7f994c97b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:12:27.932 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:27.931+0000 7f994c97b640 1 -- 172.21.15.66:0/3392241929 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9948137810 con 0x7f994813f710 2024-03-25T18:12:28.241 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.240+0000 7f9936ffd640 1 -- 172.21.15.66:0/3392241929 <== mon.0 v1:172.21.15.66:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f993c0047e0 con 0x7f994813f710 2024-03-25T18:12:28.241 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.240+0000 7f9936ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:28.241 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.240+0000 7f9936ffd640 1 -- 172.21.15.66:0/3392241929 <== mon.0 v1:172.21.15.66:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f993c004c30 con 0x7f994813f710 2024-03-25T18:12:28.241 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.240+0000 7f9936ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:28.242 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.241+0000 7f9936ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:28.242 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.241+0000 7f9936ffd640 10 monclient: dump: 2024-03-25T18:12:28.242 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:28.242 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:28.242 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:28.242 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:28.242 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:28.242 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:28.242 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:28.242 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:28.242 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:28.242 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:28.242 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.241+0000 7f9944d99640 4 set_mon_vals no callback set 2024-03-25T18:12:28.243 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.242+0000 7f9936ffd640 1 -- 172.21.15.66:0/3392241929 <== mon.0 v1:172.21.15.66:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f993c004e50 con 0x7f994813f710 2024-03-25T18:12:28.243 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.242+0000 7f994659c640 10 monclient: get_auth_request con 0x7f992806dfb0 auth_method 0 2024-03-25T18:12:28.243 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.243+0000 7f9936ffd640 10 client.4611.objecter ms_handle_connect 0x7f992806dfb0 2024-03-25T18:12:28.252 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.251+0000 7f9936ffd640 1 -- 172.21.15.66:0/3392241929 <== mon.0 v1:172.21.15.66:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f993c054810 con 0x7f994813f710 2024-03-25T18:12:28.252 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.251+0000 7f9936ffd640 10 client.4611.objecter ms_dispatch 0x7f9948000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:28.252 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.251+0000 7f9936ffd640 3 client.4611.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:28.252 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.251+0000 7f9936ffd640 3 client.4611.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:28.252 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.251+0000 7f9936ffd640 20 client.4611.objecter dump_active .. 0 homeless 2024-03-25T18:12:28.252 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.251+0000 7f9936ffd640 1 -- 172.21.15.66:0/3392241929 <== mon.0 v1:172.21.15.66:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f993c050320 con 0x7f994813f710 2024-03-25T18:12:28.252 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.251+0000 7f9936ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:28.252 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.251+0000 7f9936ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:28.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.500+0000 7f994c97b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:28.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.500+0000 7f994c97b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:28.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.500+0000 7f994c97b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:12:28.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.500+0000 7f994c97b640 1 -- 172.21.15.66:0/3392241929 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9948137c60 con 0x7f994813f710 2024-03-25T18:12:28.526 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.525+0000 7f9936ffd640 1 -- 172.21.15.66:0/3392241929 <== mon.0 v1:172.21.15.66:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f9948137c60 con 0x7f994813f710 2024-03-25T18:12:28.526 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.525+0000 7f9936ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:28.526 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.525+0000 7f9936ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:28.526 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:12:28.526 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:28.529 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.528+0000 7f994c97b640 10 librados: watch_flush enter 2024-03-25T18:12:28.529 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.528+0000 7f994c97b640 10 librados: watch_flush exit 2024-03-25T18:12:28.529 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.528+0000 7f994c97b640 20 client.4611.objecter shutdown clearing up homeless session... 2024-03-25T18:12:28.529 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.528+0000 7f994c97b640 10 client.4611.objecter successfully canceled tick 2024-03-25T18:12:28.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.528+0000 7f994c97b640 1 -- 172.21.15.66:0/3392241929 >> v1:172.21.15.193:6800/2275676803 conn(0x7f992806dfb0 legacy=0x7f9928070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:28.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.528+0000 7f994c97b640 10 monclient: shutdown 2024-03-25T18:12:28.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.528+0000 7f994c97b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:28.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.528+0000 7f994c97b640 1 -- 172.21.15.66:0/3392241929 >> v1:172.21.15.66:6789/0 conn(0x7f994813f710 legacy=0x7f99481df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:28.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.528+0000 7f994c97b640 1 -- 172.21.15.66:0/3392241929 shutdown_connections 2024-03-25T18:12:28.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.528+0000 7f994c97b640 1 -- 172.21.15.66:0/3392241929 >> 172.21.15.66:0/3392241929 conn(0x7f99480b7ea0 msgr2=0x7f9948074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:28.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.528+0000 7f994c97b640 1 -- 172.21.15.66:0/3392241929 shutdown_connections 2024-03-25T18:12:28.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.528+0000 7f994c97b640 1 -- 172.21.15.66:0/3392241929 wait complete. 2024-03-25T18:12:28.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:28.529+0000 7f994c97b640 1 librados: shutdown 2024-03-25T18:12:28.541 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:30.157 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:09:01.137 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:30.163 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:09:01.144 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:30.199 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:09:01.180 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:30.217 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:09:01.176 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:30.233 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:09:01.191 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:30.237 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:09:01.218 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:30.247 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:09:01.227 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:30.256 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:09:01.215 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:30.276 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:09:01.256 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:35.347 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:35.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.466+0000 7f0f27b3e640 10 monclient: get_monmap_and_config 2024-03-25T18:12:35.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.466+0000 7f0f27b3e640 10 monclient: build_initial_monmap 2024-03-25T18:12:35.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.466+0000 7f0f27b3e640 10 monclient: monmap: 2024-03-25T18:12:35.467 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:12:35.467 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:35.467 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:12:35.467130+0000 2024-03-25T18:12:35.467 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:12:35.467130+0000 2024-03-25T18:12:35.467 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:35.467 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:35.467 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:12:35.467 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:12:35.468 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:12:35.468 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:35.469 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.467+0000 7f0f27b3e640 1 Processor -- start 2024-03-25T18:12:35.469 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.467+0000 7f0f27b3e640 1 -- start start 2024-03-25T18:12:35.469 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.467+0000 7f0f27b3e640 10 monclient: init 2024-03-25T18:12:35.469 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.468+0000 7f0f27b3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:35.469 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.468+0000 7f0f27b3e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:12:35.469 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.468+0000 7f0f27b3e640 10 monclient(hunting): picked mon.noname-c con 0x7f0f200b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T18:12:35.469 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.468+0000 7f0f27b3e640 10 monclient(hunting): picked mon.noname-a con 0x7f0f200b1730 addr v1:172.21.15.66:6789/0 2024-03-25T18:12:35.469 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.468+0000 7f0f27b3e640 10 monclient(hunting): picked mon.noname-b con 0x7f0f2014be90 addr v1:172.21.15.66:6790/0 2024-03-25T18:12:35.469 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.468+0000 7f0f27b3e640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f200af940 con 0x7f0f200b1730 2024-03-25T18:12:35.469 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.468+0000 7f0f27b3e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f200b0a70 con 0x7f0f200b4d60 2024-03-25T18:12:35.469 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.468+0000 7f0f27b3e640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f200b0d50 con 0x7f0f2014be90 2024-03-25T18:12:35.470 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.468+0000 7f0f27b3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:35.470 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.468+0000 7f0f27b3e640 10 monclient(hunting): authenticate will time out at 7504.527332s 2024-03-25T18:12:35.483 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.482+0000 7f0f258da640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f0f200b4d60 0x7f0f2013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:60774/0 (socket says 172.21.15.66:60774) 2024-03-25T18:12:35.483 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.482+0000 7f0f258da640 1 -- 172.21.15.66:0/2157967577 learned_addr learned my addr 172.21.15.66:0/2157967577 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:12:35.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.495+0000 7f0f248d8640 1 -- 172.21.15.66:0/2157967577 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2797070681 0 0) 0x7f0f200b0a70 con 0x7f0f200b4d60 2024-03-25T18:12:35.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.495+0000 7f0f248d8640 10 monclient(hunting): _negotiate 2024-03-25T18:12:35.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.495+0000 7f0f248d8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:35.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.495+0000 7f0f248d8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:35.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.495+0000 7f0f248d8640 10 monclient(hunting): my global_id is 4609 2024-03-25T18:12:35.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.495+0000 7f0f248d8640 1 -- 172.21.15.66:0/2157967577 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f0c003520 con 0x7f0f200b4d60 2024-03-25T18:12:35.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.497+0000 7f0f248d8640 1 -- 172.21.15.66:0/2157967577 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1838640323 0 0) 0x7f0f200af940 con 0x7f0f200b1730 2024-03-25T18:12:35.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.497+0000 7f0f248d8640 10 monclient(hunting): _negotiate 2024-03-25T18:12:35.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.497+0000 7f0f248d8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:35.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.497+0000 7f0f248d8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:35.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.497+0000 7f0f248d8640 10 monclient(hunting): my global_id is 4614 2024-03-25T18:12:35.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.497+0000 7f0f248d8640 1 -- 172.21.15.66:0/2157967577 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f200b0a70 con 0x7f0f200b1730 2024-03-25T18:12:35.504 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.504+0000 7f0f248d8640 1 -- 172.21.15.66:0/2157967577 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2904456575 0 0) 0x7f0f200b0d50 con 0x7f0f2014be90 2024-03-25T18:12:35.504 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.504+0000 7f0f248d8640 10 monclient(hunting): _negotiate 2024-03-25T18:12:35.504 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.504+0000 7f0f248d8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:35.504 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.504+0000 7f0f248d8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:35.504 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.504+0000 7f0f248d8640 10 monclient(hunting): my global_id is 4610 2024-03-25T18:12:35.504 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.504+0000 7f0f248d8640 1 -- 172.21.15.66:0/2157967577 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f200af940 con 0x7f0f2014be90 2024-03-25T18:12:35.518 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.517+0000 7f0f248d8640 1 -- 172.21.15.66:0/2157967577 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4098467633 0 0) 0x7f0f0c003520 con 0x7f0f200b4d60 2024-03-25T18:12:35.518 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.517+0000 7f0f248d8640 1 -- 172.21.15.66:0/2157967577 >> v1:172.21.15.66:6790/0 conn(0x7f0f2014be90 legacy=0x7f0f2014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:35.518 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.517+0000 7f0f248d8640 1 -- 172.21.15.66:0/2157967577 >> v1:172.21.15.66:6789/0 conn(0x7f0f200b1730 legacy=0x7f0f200b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:35.518 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.517+0000 7f0f248d8640 10 monclient: _finish_hunting 0 2024-03-25T18:12:35.518 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.517+0000 7f0f248d8640 1 monclient: found mon.noname-c 2024-03-25T18:12:35.518 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.517+0000 7f0f248d8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:35.518 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.517+0000 7f0f248d8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:12:35.518 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.517+0000 7f0f248d8640 1 -- 172.21.15.66:0/2157967577 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f200b1030 con 0x7f0f200b4d60 2024-03-25T18:12:35.518 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.517+0000 7f0f248d8640 10 monclient: _finish_auth 0 2024-03-25T18:12:35.518 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.517+0000 7f0f248d8640 10 monclient: _check_auth_tickets 2024-03-25T18:12:35.519 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.517+0000 7f0f248d8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:35.519 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.517+0000 7f0f248d8640 1 -- 172.21.15.66:0/2157967577 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f0f1c001690 con 0x7f0f200b4d60 2024-03-25T18:12:35.519 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.517+0000 7f0f248d8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:35.519 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.517+0000 7f0f248d8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:35.519 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.517+0000 7f0f248d8640 10 monclient: dump: 2024-03-25T18:12:35.519 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:35.519 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:35.519 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:35.519 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:35.519 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:35.519 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:35.519 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:35.519 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:35.519 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:35.519 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:35.519 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.517+0000 7f0f27b3e640 5 monclient: authenticate success, global_id 4609 2024-03-25T18:12:35.520 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.517+0000 7f0f27b3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:35.524 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.524+0000 7f0f248d8640 1 -- 172.21.15.66:0/2157967577 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f1c0017f0 con 0x7f0f200b4d60 2024-03-25T18:12:35.524 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.524+0000 7f0f248d8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:35.524 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.524+0000 7f0f248d8640 1 -- 172.21.15.66:0/2157967577 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f0f1c001c00 con 0x7f0f200b4d60 2024-03-25T18:12:35.525 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.524+0000 7f0f27b3e640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:35.525 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.524+0000 7f0f27b3e640 4 set_mon_vals no callback set 2024-03-25T18:12:35.525 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.524+0000 7f0f248d8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:35.525 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.524+0000 7f0f248d8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:35.525 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.524+0000 7f0f248d8640 10 monclient: dump: 2024-03-25T18:12:35.525 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:35.525 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:35.525 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:35.525 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:35.525 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:35.525 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:35.525 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:35.525 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:35.525 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:35.526 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:35.528 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.525+0000 7f0f27b3e640 10 monclient: shutdown 2024-03-25T18:12:35.528 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.525+0000 7f0f27b3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:35.528 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.525+0000 7f0f27b3e640 1 -- 172.21.15.66:0/2157967577 >> v1:172.21.15.193:6789/0 conn(0x7f0f200b4d60 legacy=0x7f0f2013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:35.528 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.525+0000 7f0f27b3e640 1 -- 172.21.15.66:0/2157967577 shutdown_connections 2024-03-25T18:12:35.528 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.525+0000 7f0f27b3e640 1 -- 172.21.15.66:0/2157967577 >> 172.21.15.66:0/2157967577 conn(0x7f0f200617a0 msgr2=0x7f0f20061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:35.528 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.525+0000 7f0f27b3e640 1 -- 172.21.15.66:0/2157967577 shutdown_connections 2024-03-25T18:12:35.528 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.525+0000 7f0f27b3e640 1 -- 172.21.15.66:0/2157967577 wait complete. 2024-03-25T18:12:35.528 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.526+0000 7f0f27b3e640 10 monclient: build_initial_monmap 2024-03-25T18:12:35.528 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.526+0000 7f0f27b3e640 10 monclient: monmap: 2024-03-25T18:12:35.528 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:12:35.529 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:35.529 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:12:35.529 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:12:35.529 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:35.529 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:12:35.529 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:12:35.529 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:12:35.529 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:12:35.529 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:35.529 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.526+0000 7f0f27b3e640 1 librados: starting msgr at 2024-03-25T18:12:35.529 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.526+0000 7f0f27b3e640 1 librados: starting objecter 2024-03-25T18:12:35.529 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.526+0000 7f0f27b3e640 1 Processor -- start 2024-03-25T18:12:35.529 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.526+0000 7f0f27b3e640 1 -- start start 2024-03-25T18:12:35.529 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.526+0000 7f0f27b3e640 1 librados: setting wanted keys 2024-03-25T18:12:35.529 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.526+0000 7f0f27b3e640 1 librados: calling monclient init 2024-03-25T18:12:35.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.526+0000 7f0f27b3e640 10 monclient: init 2024-03-25T18:12:35.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.527+0000 7f0f27b3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:35.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.527+0000 7f0f27b3e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:12:35.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.527+0000 7f0f27b3e640 10 monclient(hunting): picked mon.noname-a con 0x7f0f200b1730 addr v1:172.21.15.66:6789/0 2024-03-25T18:12:35.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.527+0000 7f0f27b3e640 10 monclient(hunting): picked mon.noname-c con 0x7f0f200b4d60 addr v1:172.21.15.66:6790/0 2024-03-25T18:12:35.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.527+0000 7f0f27b3e640 10 monclient(hunting): picked mon.noname-b con 0x7f0f2014be90 addr v1:172.21.15.193:6789/0 2024-03-25T18:12:35.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.527+0000 7f0f27b3e640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f201d7940 con 0x7f0f200b1730 2024-03-25T18:12:35.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.527+0000 7f0f27b3e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f201382f0 con 0x7f0f2014be90 2024-03-25T18:12:35.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.527+0000 7f0f27b3e640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f20137320 con 0x7f0f200b4d60 2024-03-25T18:12:35.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.527+0000 7f0f27b3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:35.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.527+0000 7f0f27b3e640 10 monclient(hunting): authenticate will time out at 7504.586649s 2024-03-25T18:12:35.545 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.545+0000 7f0f250d9640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f0f200b4d60 0x7f0f201d7e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:51630/0 (socket says 172.21.15.66:51630) 2024-03-25T18:12:35.546 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.545+0000 7f0f250d9640 1 -- 172.21.15.66:0/1705766741 learned_addr learned my addr 172.21.15.66:0/1705766741 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:12:35.550 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.549+0000 7f0f167fc640 10 client.?.objecter ms_handle_connect 0x7f0f200b1730 2024-03-25T18:12:35.550 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.549+0000 7f0f167fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:35.553 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.552+0000 7f0f167fc640 10 client.?.objecter ms_handle_connect 0x7f0f200b4d60 2024-03-25T18:12:35.553 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.552+0000 7f0f167fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:35.557 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.557+0000 7f0f167fc640 10 client.?.objecter ms_handle_connect 0x7f0f2014be90 2024-03-25T18:12:35.557 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.557+0000 7f0f167fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:35.564 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.564+0000 7f0f167fc640 1 -- 172.21.15.66:0/1705766741 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2192676101 0 0) 0x7f0f201d7940 con 0x7f0f200b1730 2024-03-25T18:12:35.564 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.564+0000 7f0f167fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:35.564 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.564+0000 7f0f167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:35.565 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.564+0000 7f0f167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:35.565 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.564+0000 7f0f167fc640 10 monclient(hunting): my global_id is 4617 2024-03-25T18:12:35.565 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.564+0000 7f0f167fc640 1 -- 172.21.15.66:0/1705766741 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f04004eb0 con 0x7f0f200b1730 2024-03-25T18:12:35.566 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.566+0000 7f0f167fc640 1 -- 172.21.15.66:0/1705766741 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 771510144 0 0) 0x7f0f20137320 con 0x7f0f200b4d60 2024-03-25T18:12:35.566 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.566+0000 7f0f167fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:35.566 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.566+0000 7f0f167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:35.566 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.566+0000 7f0f167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:35.566 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.566+0000 7f0f167fc640 10 monclient(hunting): my global_id is 4613 2024-03-25T18:12:35.567 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.566+0000 7f0f167fc640 1 -- 172.21.15.66:0/1705766741 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f201d7940 con 0x7f0f200b4d60 2024-03-25T18:12:35.567 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.566+0000 7f0f167fc640 1 -- 172.21.15.66:0/1705766741 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2387694856 0 0) 0x7f0f201382f0 con 0x7f0f2014be90 2024-03-25T18:12:35.567 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.566+0000 7f0f167fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:35.567 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.566+0000 7f0f167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:35.567 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.566+0000 7f0f167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:35.567 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.566+0000 7f0f167fc640 10 monclient(hunting): my global_id is 4612 2024-03-25T18:12:35.567 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.566+0000 7f0f167fc640 1 -- 172.21.15.66:0/1705766741 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f20137320 con 0x7f0f2014be90 2024-03-25T18:12:35.582 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.582+0000 7f0f167fc640 1 -- 172.21.15.66:0/1705766741 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3622167255 0 0) 0x7f0f04004eb0 con 0x7f0f200b1730 2024-03-25T18:12:35.583 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.582+0000 7f0f167fc640 1 -- 172.21.15.66:0/1705766741 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f201382f0 con 0x7f0f200b1730 2024-03-25T18:12:35.583 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.582+0000 7f0f167fc640 1 -- 172.21.15.66:0/1705766741 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f0f1c001b00 con 0x7f0f200b1730 2024-03-25T18:12:35.583 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.582+0000 7f0f167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:35.583 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.582+0000 7f0f167fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:12:35.583 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.582+0000 7f0f167fc640 10 monclient(hunting): dump: 2024-03-25T18:12:35.583 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:35.583 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:35.583 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:35.583 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:35.583 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:35.583 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:35.583 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:35.583 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:35.583 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:35.583 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:35.583 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.582+0000 7f0f167fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:35.589 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.588+0000 7f0f167fc640 1 -- 172.21.15.66:0/1705766741 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3529698351 0 0) 0x7f0f20137320 con 0x7f0f2014be90 2024-03-25T18:12:35.589 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.588+0000 7f0f167fc640 1 -- 172.21.15.66:0/1705766741 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f04004eb0 con 0x7f0f2014be90 2024-03-25T18:12:35.589 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.588+0000 7f0f167fc640 1 -- 172.21.15.66:0/1705766741 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f0f18001b10 con 0x7f0f2014be90 2024-03-25T18:12:35.589 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.589+0000 7f0f167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:35.589 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.589+0000 7f0f167fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:35.589 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.589+0000 7f0f167fc640 10 monclient(hunting): dump: 2024-03-25T18:12:35.589 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:35.590 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:35.590 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:35.590 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:35.590 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:35.590 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:35.590 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:35.590 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:35.590 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:35.590 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:35.597 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.596+0000 7f0f167fc640 1 -- 172.21.15.66:0/1705766741 <== mon.0 v1:172.21.15.66:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 35235501 0 0) 0x7f0f201382f0 con 0x7f0f200b1730 2024-03-25T18:12:35.597 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.596+0000 7f0f167fc640 1 -- 172.21.15.66:0/1705766741 >> v1:172.21.15.66:6790/0 conn(0x7f0f200b4d60 legacy=0x7f0f201d7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:35.597 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.596+0000 7f0f167fc640 1 -- 172.21.15.66:0/1705766741 >> v1:172.21.15.193:6789/0 conn(0x7f0f2014be90 legacy=0x7f0f2012fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:35.597 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.596+0000 7f0f167fc640 10 monclient: _finish_hunting 0 2024-03-25T18:12:35.598 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.596+0000 7f0f167fc640 1 monclient: found mon.a 2024-03-25T18:12:35.598 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.596+0000 7f0f167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:35.598 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.596+0000 7f0f167fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:12:35.598 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.596+0000 7f0f167fc640 1 -- 172.21.15.66:0/1705766741 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f201d8530 con 0x7f0f200b1730 2024-03-25T18:12:35.598 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.596+0000 7f0f167fc640 10 monclient: _finish_auth 0 2024-03-25T18:12:35.598 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.596+0000 7f0f167fc640 10 monclient: _check_auth_tickets 2024-03-25T18:12:35.598 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.596+0000 7f0f167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:35.598 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.596+0000 7f0f27b3e640 5 monclient: authenticate success, global_id 4617 2024-03-25T18:12:35.598 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.597+0000 7f0f27b3e640 10 monclient: _renew_subs 2024-03-25T18:12:35.598 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.597+0000 7f0f27b3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:12:35.598 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.597+0000 7f0f27b3e640 1 -- 172.21.15.66:0/1705766741 --> v1:172.21.15.66:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f20130570 con 0x7f0f200b1730 2024-03-25T18:12:35.598 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.597+0000 7f0f27b3e640 10 client.4617.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:35.598 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.597+0000 7f0f27b3e640 10 monclient: _renew_subs 2024-03-25T18:12:35.599 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.597+0000 7f0f27b3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:12:35.599 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.597+0000 7f0f27b3e640 1 -- 172.21.15.66:0/1705766741 --> v1:172.21.15.66:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f20143ae0 con 0x7f0f200b1730 2024-03-25T18:12:35.599 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.597+0000 7f0f27b3e640 1 librados: init done 2024-03-25T18:12:35.599 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.598+0000 7f0f27b3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:35.599 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.598+0000 7f0f27b3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:35.599 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.598+0000 7f0f27b3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:12:35.599 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.598+0000 7f0f27b3e640 1 -- 172.21.15.66:0/1705766741 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f201d77a0 con 0x7f0f200b1730 2024-03-25T18:12:35.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.866+0000 7f0f167fc640 1 -- 172.21.15.66:0/1705766741 <== mon.0 v1:172.21.15.66:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f1c001220 con 0x7f0f200b1730 2024-03-25T18:12:35.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.866+0000 7f0f167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:35.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.866+0000 7f0f167fc640 1 -- 172.21.15.66:0/1705766741 <== mon.0 v1:172.21.15.66:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f0f1c004a90 con 0x7f0f200b1730 2024-03-25T18:12:35.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.866+0000 7f0f167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:35.868 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.866+0000 7f0f167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:35.868 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.866+0000 7f0f167fc640 10 monclient: dump: 2024-03-25T18:12:35.868 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:35.868 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:35.868 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:35.868 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:35.868 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:35.868 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:35.868 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:35.868 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:35.868 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:35.868 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:35.868 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.866+0000 7f0f17fff640 4 set_mon_vals no callback set 2024-03-25T18:12:35.869 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.867+0000 7f0f167fc640 1 -- 172.21.15.66:0/1705766741 <== mon.0 v1:172.21.15.66:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f0f1c01aa60 con 0x7f0f200b1730 2024-03-25T18:12:35.869 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.868+0000 7f0f250d9640 10 monclient: get_auth_request con 0x7f0f0406df10 auth_method 0 2024-03-25T18:12:35.869 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.868+0000 7f0f167fc640 10 client.4617.objecter ms_handle_connect 0x7f0f0406df10 2024-03-25T18:12:35.873 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.872+0000 7f0f167fc640 1 -- 172.21.15.66:0/1705766741 <== mon.0 v1:172.21.15.66:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f0f1c054700 con 0x7f0f200b1730 2024-03-25T18:12:35.873 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.872+0000 7f0f167fc640 10 client.4617.objecter ms_dispatch 0x7f0f20000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:35.873 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.872+0000 7f0f167fc640 3 client.4617.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:35.873 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.872+0000 7f0f167fc640 3 client.4617.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:35.873 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.872+0000 7f0f167fc640 20 client.4617.objecter dump_active .. 0 homeless 2024-03-25T18:12:35.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.872+0000 7f0f167fc640 1 -- 172.21.15.66:0/1705766741 <== mon.0 v1:172.21.15.66:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0f1c050040 con 0x7f0f200b1730 2024-03-25T18:12:35.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.872+0000 7f0f167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:35.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:35.872+0000 7f0f167fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:36.113 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:36.113+0000 7f0f27b3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:36.114 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:36.113+0000 7f0f27b3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:36.114 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:36.113+0000 7f0f27b3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:12:36.114 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:36.113+0000 7f0f27b3e640 1 -- 172.21.15.66:0/1705766741 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f201d86c0 con 0x7f0f200b1730 2024-03-25T18:12:36.160 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:36.160+0000 7f0f167fc640 1 -- 172.21.15.66:0/1705766741 <== mon.0 v1:172.21.15.66:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f0f201d86c0 con 0x7f0f200b1730 2024-03-25T18:12:36.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:36.160+0000 7f0f167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:36.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:36.160+0000 7f0f167fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:36.161 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:12:36.161 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:36.163 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:36.162+0000 7f0f27b3e640 10 librados: watch_flush enter 2024-03-25T18:12:36.163 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:36.162+0000 7f0f27b3e640 10 librados: watch_flush exit 2024-03-25T18:12:36.163 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:36.162+0000 7f0f27b3e640 20 client.4617.objecter shutdown clearing up homeless session... 2024-03-25T18:12:36.163 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:36.162+0000 7f0f27b3e640 10 client.4617.objecter successfully canceled tick 2024-03-25T18:12:36.164 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:36.162+0000 7f0f27b3e640 1 -- 172.21.15.66:0/1705766741 >> v1:172.21.15.193:6800/2275676803 conn(0x7f0f0406df10 legacy=0x7f0f040703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:36.164 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:36.162+0000 7f0f27b3e640 10 monclient: shutdown 2024-03-25T18:12:36.164 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:36.162+0000 7f0f27b3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:36.164 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:36.162+0000 7f0f27b3e640 1 -- 172.21.15.66:0/1705766741 >> v1:172.21.15.66:6789/0 conn(0x7f0f200b1730 legacy=0x7f0f20136c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:36.164 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:36.162+0000 7f0f27b3e640 1 -- 172.21.15.66:0/1705766741 shutdown_connections 2024-03-25T18:12:36.164 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:36.162+0000 7f0f27b3e640 1 -- 172.21.15.66:0/1705766741 >> 172.21.15.66:0/1705766741 conn(0x7f0f200617a0 msgr2=0x7f0f200b2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:36.164 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:36.162+0000 7f0f27b3e640 1 -- 172.21.15.66:0/1705766741 shutdown_connections 2024-03-25T18:12:36.164 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:36.162+0000 7f0f27b3e640 1 -- 172.21.15.66:0/1705766741 wait complete. 2024-03-25T18:12:36.164 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:36.162+0000 7f0f27b3e640 1 librados: shutdown 2024-03-25T18:12:36.175 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:42.982 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:43.102 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.101+0000 7f549fe3e640 10 monclient: get_monmap_and_config 2024-03-25T18:12:43.103 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.101+0000 7f549fe3e640 10 monclient: build_initial_monmap 2024-03-25T18:12:43.103 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.101+0000 7f549fe3e640 10 monclient: monmap: 2024-03-25T18:12:43.103 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:12:43.103 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:43.103 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:12:43.102823+0000 2024-03-25T18:12:43.103 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:12:43.102823+0000 2024-03-25T18:12:43.103 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:43.103 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:43.103 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:12:43.103 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:12:43.103 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:12:43.103 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:43.105 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.102+0000 7f549fe3e640 1 Processor -- start 2024-03-25T18:12:43.105 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.102+0000 7f549fe3e640 1 -- start start 2024-03-25T18:12:43.105 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.102+0000 7f549fe3e640 10 monclient: init 2024-03-25T18:12:43.105 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.103+0000 7f549fe3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:43.105 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.103+0000 7f549fe3e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:12:43.105 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.103+0000 7f549fe3e640 10 monclient(hunting): picked mon.noname-c con 0x7f54980b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T18:12:43.106 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.103+0000 7f549fe3e640 10 monclient(hunting): picked mon.noname-b con 0x7f54980b1730 addr v1:172.21.15.66:6790/0 2024-03-25T18:12:43.106 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.103+0000 7f549fe3e640 10 monclient(hunting): picked mon.noname-a con 0x7f549814be90 addr v1:172.21.15.66:6789/0 2024-03-25T18:12:43.106 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.103+0000 7f549fe3e640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54980af940 con 0x7f549814be90 2024-03-25T18:12:43.106 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.103+0000 7f549fe3e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54980b0a70 con 0x7f54980b4d60 2024-03-25T18:12:43.106 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.103+0000 7f549fe3e640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54980b0d50 con 0x7f54980b1730 2024-03-25T18:12:43.106 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.103+0000 7f549fe3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:43.106 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.103+0000 7f549fe3e640 10 monclient(hunting): authenticate will time out at 7512.163642s 2024-03-25T18:12:43.111 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.109+0000 7f549dbda640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f54980b4d60 0x7f549813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:36804/0 (socket says 172.21.15.66:36804) 2024-03-25T18:12:43.111 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.109+0000 7f549dbda640 1 -- 172.21.15.66:0/1438466575 learned_addr learned my addr 172.21.15.66:0/1438466575 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:12:43.120 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.118+0000 7f549cbd8640 1 -- 172.21.15.66:0/1438466575 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 748521001 0 0) 0x7f54980b0a70 con 0x7f54980b4d60 2024-03-25T18:12:43.120 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.118+0000 7f549cbd8640 10 monclient(hunting): _negotiate 2024-03-25T18:12:43.120 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.118+0000 7f549cbd8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:43.120 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.118+0000 7f549cbd8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:43.120 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.118+0000 7f549cbd8640 10 monclient(hunting): my global_id is 4615 2024-03-25T18:12:43.120 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.118+0000 7f549cbd8640 1 -- 172.21.15.66:0/1438466575 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5484003520 con 0x7f54980b4d60 2024-03-25T18:12:43.129 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.127+0000 7f549cbd8640 1 -- 172.21.15.66:0/1438466575 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1595109268 0 0) 0x7f5484003520 con 0x7f54980b4d60 2024-03-25T18:12:43.129 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.127+0000 7f549cbd8640 1 -- 172.21.15.66:0/1438466575 >> v1:172.21.15.66:6790/0 conn(0x7f54980b1730 legacy=0x7f54980b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:43.129 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.127+0000 7f549cbd8640 1 -- 172.21.15.66:0/1438466575 >> v1:172.21.15.66:6789/0 conn(0x7f549814be90 legacy=0x7f549814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:43.129 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.127+0000 7f549cbd8640 10 monclient: _finish_hunting 0 2024-03-25T18:12:43.129 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.127+0000 7f549cbd8640 1 monclient: found mon.noname-c 2024-03-25T18:12:43.129 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.127+0000 7f549cbd8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:43.129 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.127+0000 7f549cbd8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:12:43.129 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.127+0000 7f549cbd8640 1 -- 172.21.15.66:0/1438466575 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54980b1030 con 0x7f54980b4d60 2024-03-25T18:12:43.130 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.127+0000 7f549cbd8640 10 monclient: _finish_auth 0 2024-03-25T18:12:43.130 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.127+0000 7f549cbd8640 10 monclient: _check_auth_tickets 2024-03-25T18:12:43.130 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.127+0000 7f549cbd8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:43.130 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.127+0000 7f549cbd8640 1 -- 172.21.15.66:0/1438466575 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f5494001690 con 0x7f54980b4d60 2024-03-25T18:12:43.130 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.127+0000 7f549cbd8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:43.130 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.127+0000 7f549cbd8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:43.130 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.127+0000 7f549cbd8640 10 monclient: dump: 2024-03-25T18:12:43.130 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:43.130 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:43.130 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:43.130 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:43.130 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:43.130 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:43.130 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:43.131 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:43.131 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:43.131 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:43.131 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.127+0000 7f549fe3e640 5 monclient: authenticate success, global_id 4615 2024-03-25T18:12:43.131 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.127+0000 7f549fe3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:43.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.136+0000 7f549cbd8640 1 -- 172.21.15.66:0/1438466575 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54940017f0 con 0x7f54980b4d60 2024-03-25T18:12:43.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.136+0000 7f549cbd8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:43.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.136+0000 7f549cbd8640 1 -- 172.21.15.66:0/1438466575 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f5494001c00 con 0x7f54980b4d60 2024-03-25T18:12:43.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.136+0000 7f549cbd8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:43.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.136+0000 7f549cbd8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:43.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.136+0000 7f549cbd8640 10 monclient: dump: 2024-03-25T18:12:43.138 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:43.138 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:43.138 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:43.138 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:43.138 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:43.138 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:43.138 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:43.138 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:43.139 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:43.139 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:43.139 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.136+0000 7f549fe3e640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:43.139 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.136+0000 7f549fe3e640 4 set_mon_vals no callback set 2024-03-25T18:12:43.140 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.137+0000 7f549fe3e640 10 monclient: shutdown 2024-03-25T18:12:43.140 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.137+0000 7f549fe3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:43.141 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.137+0000 7f549fe3e640 1 -- 172.21.15.66:0/1438466575 >> v1:172.21.15.193:6789/0 conn(0x7f54980b4d60 legacy=0x7f549813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:43.141 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.137+0000 7f549fe3e640 1 -- 172.21.15.66:0/1438466575 shutdown_connections 2024-03-25T18:12:43.141 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.137+0000 7f549fe3e640 1 -- 172.21.15.66:0/1438466575 >> 172.21.15.66:0/1438466575 conn(0x7f54980617a0 msgr2=0x7f5498061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:43.141 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.137+0000 7f549fe3e640 1 -- 172.21.15.66:0/1438466575 shutdown_connections 2024-03-25T18:12:43.141 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.137+0000 7f549fe3e640 1 -- 172.21.15.66:0/1438466575 wait complete. 2024-03-25T18:12:43.141 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.138+0000 7f549fe3e640 10 monclient: build_initial_monmap 2024-03-25T18:12:43.141 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.138+0000 7f549fe3e640 10 monclient: monmap: 2024-03-25T18:12:43.141 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:12:43.141 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:43.141 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:12:43.141 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:12:43.141 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:43.141 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:12:43.141 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:12:43.141 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:12:43.142 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:12:43.142 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:43.142 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.139+0000 7f549fe3e640 1 librados: starting msgr at 2024-03-25T18:12:43.142 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.139+0000 7f549fe3e640 1 librados: starting objecter 2024-03-25T18:12:43.142 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.139+0000 7f549fe3e640 1 Processor -- start 2024-03-25T18:12:43.142 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.139+0000 7f549fe3e640 1 -- start start 2024-03-25T18:12:43.142 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.139+0000 7f549fe3e640 1 librados: setting wanted keys 2024-03-25T18:12:43.142 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.139+0000 7f549fe3e640 1 librados: calling monclient init 2024-03-25T18:12:43.142 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.139+0000 7f549fe3e640 10 monclient: init 2024-03-25T18:12:43.143 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.140+0000 7f549fe3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:43.143 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.140+0000 7f549fe3e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:12:43.143 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.140+0000 7f549fe3e640 10 monclient(hunting): picked mon.noname-c con 0x7f54980b1730 addr v1:172.21.15.66:6790/0 2024-03-25T18:12:43.143 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.140+0000 7f549fe3e640 10 monclient(hunting): picked mon.noname-a con 0x7f54980b4d60 addr v1:172.21.15.66:6789/0 2024-03-25T18:12:43.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.140+0000 7f549fe3e640 10 monclient(hunting): picked mon.noname-b con 0x7f549814be90 addr v1:172.21.15.193:6789/0 2024-03-25T18:12:43.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.140+0000 7f549fe3e640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5498141f40 con 0x7f54980b4d60 2024-03-25T18:12:43.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.140+0000 7f549fe3e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5498142360 con 0x7f549814be90 2024-03-25T18:12:43.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.140+0000 7f549fe3e640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54981424f0 con 0x7f54980b1730 2024-03-25T18:12:43.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.140+0000 7f549fe3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:43.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.140+0000 7f549fe3e640 10 monclient(hunting): authenticate will time out at 7512.200978s 2024-03-25T18:12:43.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.153+0000 7f549d3d9640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f54980b4d60 0x7f54981d7c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:44846/0 (socket says 172.21.15.66:44846) 2024-03-25T18:12:43.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.153+0000 7f549d3d9640 1 -- 172.21.15.66:0/2820951071 learned_addr learned my addr 172.21.15.66:0/2820951071 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:12:43.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.160+0000 7f548e7fc640 10 client.?.objecter ms_handle_connect 0x7f54980b1730 2024-03-25T18:12:43.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.160+0000 7f548e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:43.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.160+0000 7f548e7fc640 10 client.?.objecter ms_handle_connect 0x7f54980b4d60 2024-03-25T18:12:43.161 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.160+0000 7f548e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:43.166 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.165+0000 7f548e7fc640 1 -- 172.21.15.66:0/2820951071 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3786297421 0 0) 0x7f54980b1030 con 0x7f54980b1730 2024-03-25T18:12:43.166 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.165+0000 7f548e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:43.166 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.165+0000 7f548e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:43.166 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.165+0000 7f548e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:43.167 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.165+0000 7f548e7fc640 10 monclient(hunting): my global_id is 4619 2024-03-25T18:12:43.167 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.165+0000 7f548e7fc640 1 -- 172.21.15.66:0/2820951071 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f547c004eb0 con 0x7f54980b1730 2024-03-25T18:12:43.173 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.171+0000 7f548e7fc640 1 -- 172.21.15.66:0/2820951071 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424961454 0 0) 0x7f5498141f40 con 0x7f54980b4d60 2024-03-25T18:12:43.173 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.171+0000 7f548e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:43.173 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.171+0000 7f548e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:43.173 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.171+0000 7f548e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:43.173 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.171+0000 7f548e7fc640 10 monclient(hunting): my global_id is 4623 2024-03-25T18:12:43.173 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.171+0000 7f548e7fc640 1 -- 172.21.15.66:0/2820951071 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54980b1030 con 0x7f54980b4d60 2024-03-25T18:12:43.177 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.175+0000 7f548e7fc640 10 client.?.objecter ms_handle_connect 0x7f549814be90 2024-03-25T18:12:43.177 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.175+0000 7f548e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:43.188 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.186+0000 7f548e7fc640 1 -- 172.21.15.66:0/2820951071 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3424338234 0 0) 0x7f547c004eb0 con 0x7f54980b1730 2024-03-25T18:12:43.188 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.186+0000 7f548e7fc640 1 -- 172.21.15.66:0/2820951071 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f547c005490 con 0x7f54980b1730 2024-03-25T18:12:43.188 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.186+0000 7f548e7fc640 1 -- 172.21.15.66:0/2820951071 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f5494001960 con 0x7f54980b1730 2024-03-25T18:12:43.188 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.186+0000 7f548e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:43.188 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.186+0000 7f548e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:43.188 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.186+0000 7f548e7fc640 10 monclient(hunting): dump: 2024-03-25T18:12:43.188 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:43.188 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:43.188 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:43.189 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:43.189 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:43.189 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:43.189 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:43.189 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:43.189 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:43.189 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:43.189 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.186+0000 7f548e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:43.192 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.191+0000 7f548e7fc640 1 -- 172.21.15.66:0/2820951071 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3218463085 0 0) 0x7f5498142360 con 0x7f549814be90 2024-03-25T18:12:43.192 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.191+0000 7f548e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:43.192 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.191+0000 7f548e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:43.192 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.191+0000 7f548e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:43.192 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.191+0000 7f548e7fc640 10 monclient(hunting): my global_id is 4618 2024-03-25T18:12:43.192 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.191+0000 7f548e7fc640 1 -- 172.21.15.66:0/2820951071 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f547c004eb0 con 0x7f549814be90 2024-03-25T18:12:43.194 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.192+0000 7f548e7fc640 1 -- 172.21.15.66:0/2820951071 <== mon.2 v1:172.21.15.66:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2301672152 0 0) 0x7f547c005490 con 0x7f54980b1730 2024-03-25T18:12:43.194 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.192+0000 7f548e7fc640 1 -- 172.21.15.66:0/2820951071 >> v1:172.21.15.193:6789/0 conn(0x7f549814be90 legacy=0x7f54981e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:43.194 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.192+0000 7f548e7fc640 1 -- 172.21.15.66:0/2820951071 >> v1:172.21.15.66:6789/0 conn(0x7f54980b4d60 legacy=0x7f54981d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:43.194 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.192+0000 7f548e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:12:43.194 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.192+0000 7f548e7fc640 1 monclient: found mon.c 2024-03-25T18:12:43.194 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.192+0000 7f548e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:43.194 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.192+0000 7f548e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:12:43.194 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.192+0000 7f548e7fc640 1 -- 172.21.15.66:0/2820951071 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5498140d00 con 0x7f54980b1730 2024-03-25T18:12:43.195 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.192+0000 7f548e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:12:43.195 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.192+0000 7f548e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:12:43.195 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.192+0000 7f548e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:43.195 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.192+0000 7f549fe3e640 5 monclient: authenticate success, global_id 4619 2024-03-25T18:12:43.195 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.192+0000 7f549fe3e640 10 monclient: _renew_subs 2024-03-25T18:12:43.195 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.192+0000 7f549fe3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:12:43.195 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.192+0000 7f549fe3e640 1 -- 172.21.15.66:0/2820951071 --> v1:172.21.15.66:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5498138150 con 0x7f54980b1730 2024-03-25T18:12:43.195 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.192+0000 7f549fe3e640 10 client.4619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:43.195 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.192+0000 7f549fe3e640 10 monclient: _renew_subs 2024-03-25T18:12:43.195 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.192+0000 7f549fe3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:12:43.195 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.192+0000 7f549fe3e640 1 -- 172.21.15.66:0/2820951071 --> v1:172.21.15.66:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54981d8520 con 0x7f54980b1730 2024-03-25T18:12:43.195 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.192+0000 7f549fe3e640 1 librados: init done 2024-03-25T18:12:43.196 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.193+0000 7f549fe3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:43.196 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.193+0000 7f549fe3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:43.196 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.193+0000 7f549fe3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:12:43.196 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.193+0000 7f549fe3e640 1 -- 172.21.15.66:0/2820951071 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5498140f50 con 0x7f54980b1730 2024-03-25T18:12:43.513 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.511+0000 7f548e7fc640 1 -- 172.21.15.66:0/2820951071 <== mon.2 v1:172.21.15.66:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5494001b80 con 0x7f54980b1730 2024-03-25T18:12:43.513 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.511+0000 7f548e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:43.513 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.511+0000 7f548e7fc640 1 -- 172.21.15.66:0/2820951071 <== mon.2 v1:172.21.15.66:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f5494004c30 con 0x7f54980b1730 2024-03-25T18:12:43.513 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.511+0000 7f548e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:43.513 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.511+0000 7f548e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:43.513 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.511+0000 7f548e7fc640 10 monclient: dump: 2024-03-25T18:12:43.513 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:43.513 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:43.513 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:43.513 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:43.513 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:43.513 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:43.513 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:43.514 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:43.514 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:43.514 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:43.514 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.511+0000 7f548ffff640 4 set_mon_vals no callback set 2024-03-25T18:12:43.514 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.512+0000 7f548e7fc640 1 -- 172.21.15.66:0/2820951071 <== mon.2 v1:172.21.15.66:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f549401ac00 con 0x7f54980b1730 2024-03-25T18:12:43.514 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.513+0000 7f549d3d9640 10 monclient: get_auth_request con 0x7f547c06e050 auth_method 0 2024-03-25T18:12:43.514 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.513+0000 7f548e7fc640 10 client.4619.objecter ms_handle_connect 0x7f547c06e050 2024-03-25T18:12:43.520 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.519+0000 7f548e7fc640 1 -- 172.21.15.66:0/2820951071 <== mon.2 v1:172.21.15.66:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f5494054680 con 0x7f54980b1730 2024-03-25T18:12:43.520 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.519+0000 7f548e7fc640 10 client.4619.objecter ms_dispatch 0x7f5498000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:43.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.519+0000 7f548e7fc640 3 client.4619.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:43.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.519+0000 7f548e7fc640 3 client.4619.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:43.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.519+0000 7f548e7fc640 20 client.4619.objecter dump_active .. 0 homeless 2024-03-25T18:12:43.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.519+0000 7f548e7fc640 1 -- 172.21.15.66:0/2820951071 <== mon.2 v1:172.21.15.66:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5494050190 con 0x7f54980b1730 2024-03-25T18:12:43.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.519+0000 7f548e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:43.521 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.519+0000 7f548e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:43.785 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.783+0000 7f549fe3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:43.785 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.783+0000 7f549fe3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:43.785 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.783+0000 7f549fe3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:12:43.785 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.783+0000 7f549fe3e640 1 -- 172.21.15.66:0/2820951071 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54980627b0 con 0x7f54980b1730 2024-03-25T18:12:43.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.809+0000 7f548e7fc640 1 -- 172.21.15.66:0/2820951071 <== mon.2 v1:172.21.15.66:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f54940520c0 con 0x7f54980b1730 2024-03-25T18:12:43.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.809+0000 7f548e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:43.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.809+0000 7f548e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:43.811 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:12:43.811 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:43.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.812+0000 7f549fe3e640 10 librados: watch_flush enter 2024-03-25T18:12:43.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.812+0000 7f549fe3e640 10 librados: watch_flush exit 2024-03-25T18:12:43.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.812+0000 7f549fe3e640 20 client.4619.objecter shutdown clearing up homeless session... 2024-03-25T18:12:43.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.812+0000 7f549fe3e640 10 client.4619.objecter successfully canceled tick 2024-03-25T18:12:43.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.812+0000 7f549fe3e640 1 -- 172.21.15.66:0/2820951071 >> v1:172.21.15.193:6800/2275676803 conn(0x7f547c06e050 legacy=0x7f547c070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:43.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.812+0000 7f549fe3e640 10 monclient: shutdown 2024-03-25T18:12:43.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.812+0000 7f549fe3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:43.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.812+0000 7f549fe3e640 1 -- 172.21.15.66:0/2820951071 >> v1:172.21.15.66:6790/0 conn(0x7f54980b1730 legacy=0x7f54981405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:43.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.812+0000 7f549fe3e640 1 -- 172.21.15.66:0/2820951071 shutdown_connections 2024-03-25T18:12:43.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.812+0000 7f549fe3e640 1 -- 172.21.15.66:0/2820951071 >> 172.21.15.66:0/2820951071 conn(0x7f54980617a0 msgr2=0x7f54980b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:43.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.812+0000 7f549fe3e640 1 -- 172.21.15.66:0/2820951071 shutdown_connections 2024-03-25T18:12:43.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.812+0000 7f549fe3e640 1 -- 172.21.15.66:0/2820951071 wait complete. 2024-03-25T18:12:43.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:43.813+0000 7f549fe3e640 1 librados: shutdown 2024-03-25T18:12:43.826 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:50.244 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:09:21.224 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:50.248 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:09:21.229 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:50.291 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:09:21.271 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:50.296 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:09:21.255 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:50.300 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:09:21.258 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:50.304 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:09:21.285 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:50.315 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:09:21.296 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:50.320 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:09:21.279 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:50.345 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:09:21.326 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:50.632 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:50.752 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.750+0000 7fd99f6e4640 10 monclient: get_monmap_and_config 2024-03-25T18:12:50.752 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.750+0000 7fd99f6e4640 10 monclient: build_initial_monmap 2024-03-25T18:12:50.752 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.750+0000 7fd99f6e4640 10 monclient: monmap: 2024-03-25T18:12:50.752 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:12:50.752 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:50.752 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:12:50.751924+0000 2024-03-25T18:12:50.752 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:12:50.751924+0000 2024-03-25T18:12:50.752 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:50.752 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:50.752 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:12:50.752 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:12:50.752 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:12:50.753 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:50.753 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.751+0000 7fd99f6e4640 1 Processor -- start 2024-03-25T18:12:50.753 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.751+0000 7fd99f6e4640 1 -- start start 2024-03-25T18:12:50.753 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.751+0000 7fd99f6e4640 10 monclient: init 2024-03-25T18:12:50.754 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.752+0000 7fd99f6e4640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:50.754 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.752+0000 7fd99f6e4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:12:50.754 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.752+0000 7fd99f6e4640 10 monclient(hunting): picked mon.noname-b con 0x7fd9980b4d60 addr v1:172.21.15.66:6790/0 2024-03-25T18:12:50.754 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.752+0000 7fd99f6e4640 10 monclient(hunting): picked mon.noname-c con 0x7fd9980b1730 addr v1:172.21.15.193:6789/0 2024-03-25T18:12:50.754 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.752+0000 7fd99f6e4640 10 monclient(hunting): picked mon.noname-a con 0x7fd99814be90 addr v1:172.21.15.66:6789/0 2024-03-25T18:12:50.754 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.752+0000 7fd99f6e4640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9980af940 con 0x7fd99814be90 2024-03-25T18:12:50.754 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.752+0000 7fd99f6e4640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9980b0a70 con 0x7fd9980b1730 2024-03-25T18:12:50.755 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.752+0000 7fd99f6e4640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9980b0d50 con 0x7fd9980b4d60 2024-03-25T18:12:50.755 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.752+0000 7fd99f6e4640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:50.755 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.752+0000 7fd99f6e4640 10 monclient(hunting): authenticate will time out at 7519.812764s 2024-03-25T18:12:50.770 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.769+0000 7fd99dc81640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7fd99814be90 0x7fd99814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:55560/0 (socket says 172.21.15.66:55560) 2024-03-25T18:12:50.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.769+0000 7fd99dc81640 1 -- 172.21.15.66:0/3758017266 learned_addr learned my addr 172.21.15.66:0/3758017266 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:12:50.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.783+0000 7fd98ffff640 1 -- 172.21.15.66:0/3758017266 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4049382250 0 0) 0x7fd9980b0a70 con 0x7fd9980b1730 2024-03-25T18:12:50.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.783+0000 7fd98ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:50.785 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.783+0000 7fd98ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:50.785 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.783+0000 7fd98ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:50.785 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.783+0000 7fd98ffff640 10 monclient(hunting): my global_id is 4621 2024-03-25T18:12:50.785 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.783+0000 7fd98ffff640 1 -- 172.21.15.66:0/3758017266 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd984003520 con 0x7fd9980b1730 2024-03-25T18:12:50.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.789+0000 7fd98ffff640 1 -- 172.21.15.66:0/3758017266 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1423051383 0 0) 0x7fd9980af940 con 0x7fd99814be90 2024-03-25T18:12:50.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.789+0000 7fd98ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:50.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.789+0000 7fd98ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:50.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.789+0000 7fd98ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:50.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.789+0000 7fd98ffff640 10 monclient(hunting): my global_id is 4626 2024-03-25T18:12:50.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.789+0000 7fd98ffff640 1 -- 172.21.15.66:0/3758017266 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9980b0a70 con 0x7fd99814be90 2024-03-25T18:12:50.793 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.792+0000 7fd98ffff640 1 -- 172.21.15.66:0/3758017266 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2416113061 0 0) 0x7fd9980b0d50 con 0x7fd9980b4d60 2024-03-25T18:12:50.793 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.792+0000 7fd98ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:50.793 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.792+0000 7fd98ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:50.793 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.792+0000 7fd98ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:50.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.792+0000 7fd98ffff640 10 monclient(hunting): my global_id is 4622 2024-03-25T18:12:50.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.792+0000 7fd98ffff640 1 -- 172.21.15.66:0/3758017266 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9980af940 con 0x7fd9980b4d60 2024-03-25T18:12:50.798 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.796+0000 7fd98ffff640 1 -- 172.21.15.66:0/3758017266 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 46355974 0 0) 0x7fd984003520 con 0x7fd9980b1730 2024-03-25T18:12:50.798 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.796+0000 7fd98ffff640 1 -- 172.21.15.66:0/3758017266 >> v1:172.21.15.66:6790/0 conn(0x7fd9980b4d60 legacy=0x7fd99813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:50.798 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.796+0000 7fd98ffff640 1 -- 172.21.15.66:0/3758017266 >> v1:172.21.15.66:6789/0 conn(0x7fd99814be90 legacy=0x7fd99814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:50.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.796+0000 7fd98ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:12:50.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.796+0000 7fd98ffff640 1 monclient: found mon.noname-c 2024-03-25T18:12:50.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.796+0000 7fd98ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:50.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.796+0000 7fd98ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:12:50.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.796+0000 7fd98ffff640 1 -- 172.21.15.66:0/3758017266 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9980b1030 con 0x7fd9980b1730 2024-03-25T18:12:50.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.796+0000 7fd98ffff640 10 monclient: _finish_auth 0 2024-03-25T18:12:50.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.796+0000 7fd98ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:12:50.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.796+0000 7fd98ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:50.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.796+0000 7fd98ffff640 1 -- 172.21.15.66:0/3758017266 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fd988001690 con 0x7fd9980b1730 2024-03-25T18:12:50.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.796+0000 7fd98ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:50.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.796+0000 7fd98ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:50.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.796+0000 7fd98ffff640 10 monclient: dump: 2024-03-25T18:12:50.799 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:50.799 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:50.800 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:50.800 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:50.800 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:50.800 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:50.800 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:50.800 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:50.800 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:50.800 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:50.800 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.796+0000 7fd99f6e4640 5 monclient: authenticate success, global_id 4621 2024-03-25T18:12:50.800 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.796+0000 7fd99f6e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:50.816 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.815+0000 7fd98ffff640 1 -- 172.21.15.66:0/3758017266 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9880017f0 con 0x7fd9980b1730 2024-03-25T18:12:50.816 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.815+0000 7fd98ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:50.816 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.815+0000 7fd98ffff640 1 -- 172.21.15.66:0/3758017266 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fd988001c00 con 0x7fd9980b1730 2024-03-25T18:12:50.817 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.815+0000 7fd99f6e4640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:50.817 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.815+0000 7fd99f6e4640 4 set_mon_vals no callback set 2024-03-25T18:12:50.817 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.815+0000 7fd98ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:50.817 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.815+0000 7fd98ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:50.817 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.815+0000 7fd98ffff640 10 monclient: dump: 2024-03-25T18:12:50.817 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:50.817 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:50.817 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:50.817 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:50.817 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:50.817 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:50.817 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:50.817 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:50.817 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:50.817 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:50.819 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.815+0000 7fd99f6e4640 10 monclient: shutdown 2024-03-25T18:12:50.819 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.815+0000 7fd99f6e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:50.819 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.815+0000 7fd99f6e4640 1 -- 172.21.15.66:0/3758017266 >> v1:172.21.15.193:6789/0 conn(0x7fd9980b1730 legacy=0x7fd9980b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:50.819 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.816+0000 7fd99f6e4640 1 -- 172.21.15.66:0/3758017266 shutdown_connections 2024-03-25T18:12:50.819 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.816+0000 7fd99f6e4640 1 -- 172.21.15.66:0/3758017266 >> 172.21.15.66:0/3758017266 conn(0x7fd9980617a0 msgr2=0x7fd998061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:50.819 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.816+0000 7fd99f6e4640 1 -- 172.21.15.66:0/3758017266 shutdown_connections 2024-03-25T18:12:50.819 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.816+0000 7fd99f6e4640 1 -- 172.21.15.66:0/3758017266 wait complete. 2024-03-25T18:12:50.819 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.816+0000 7fd99f6e4640 10 monclient: build_initial_monmap 2024-03-25T18:12:50.820 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.817+0000 7fd99f6e4640 10 monclient: monmap: 2024-03-25T18:12:50.820 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:12:50.820 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:50.820 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:12:50.820 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:12:50.820 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:50.820 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:12:50.820 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:12:50.820 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:12:50.820 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:12:50.820 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:50.820 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.817+0000 7fd99f6e4640 1 librados: starting msgr at 2024-03-25T18:12:50.820 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.817+0000 7fd99f6e4640 1 librados: starting objecter 2024-03-25T18:12:50.820 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.817+0000 7fd99f6e4640 1 Processor -- start 2024-03-25T18:12:50.820 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.817+0000 7fd99f6e4640 1 -- start start 2024-03-25T18:12:50.820 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.817+0000 7fd99f6e4640 1 librados: setting wanted keys 2024-03-25T18:12:50.820 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.817+0000 7fd99f6e4640 1 librados: calling monclient init 2024-03-25T18:12:50.821 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.817+0000 7fd99f6e4640 10 monclient: init 2024-03-25T18:12:50.822 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.818+0000 7fd99f6e4640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:50.822 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.818+0000 7fd99f6e4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:12:50.822 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.818+0000 7fd99f6e4640 10 monclient(hunting): picked mon.noname-b con 0x7fd9980b1730 addr v1:172.21.15.193:6789/0 2024-03-25T18:12:50.822 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.818+0000 7fd99f6e4640 10 monclient(hunting): picked mon.noname-c con 0x7fd9980b4d60 addr v1:172.21.15.66:6790/0 2024-03-25T18:12:50.822 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.818+0000 7fd99f6e4640 10 monclient(hunting): picked mon.noname-a con 0x7fd99814be90 addr v1:172.21.15.66:6789/0 2024-03-25T18:12:50.822 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.818+0000 7fd99f6e4640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9981db590 con 0x7fd99814be90 2024-03-25T18:12:50.822 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.818+0000 7fd99f6e4640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd998141fc0 con 0x7fd9980b1730 2024-03-25T18:12:50.822 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.818+0000 7fd99f6e4640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd998140590 con 0x7fd9980b4d60 2024-03-25T18:12:50.822 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.818+0000 7fd99f6e4640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:50.822 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.818+0000 7fd99f6e4640 10 monclient(hunting): authenticate will time out at 7519.878934s 2024-03-25T18:12:50.829 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.827+0000 7fd99d480640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fd9980b1730 0x7fd9981dae80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:43014/0 (socket says 172.21.15.66:43014) 2024-03-25T18:12:50.829 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.827+0000 7fd99d480640 1 -- 172.21.15.66:0/482501852 learned_addr learned my addr 172.21.15.66:0/482501852 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:12:50.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.828+0000 7fd98dffb640 10 client.?.objecter ms_handle_connect 0x7fd9980b4d60 2024-03-25T18:12:50.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.828+0000 7fd98dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:50.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.829+0000 7fd98dffb640 10 client.?.objecter ms_handle_connect 0x7fd9980b1730 2024-03-25T18:12:50.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.829+0000 7fd98dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:50.833 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.832+0000 7fd98dffb640 10 client.?.objecter ms_handle_connect 0x7fd99814be90 2024-03-25T18:12:50.833 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.832+0000 7fd98dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:50.838 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.837+0000 7fd98dffb640 1 -- 172.21.15.66:0/482501852 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4047060484 0 0) 0x7fd998141fc0 con 0x7fd9980b1730 2024-03-25T18:12:50.839 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.837+0000 7fd98dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:12:50.839 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.837+0000 7fd98dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:50.839 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.837+0000 7fd98dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:50.839 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.837+0000 7fd98dffb640 10 monclient(hunting): my global_id is 4624 2024-03-25T18:12:50.839 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.837+0000 7fd98dffb640 1 -- 172.21.15.66:0/482501852 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd97c004eb0 con 0x7fd9980b1730 2024-03-25T18:12:50.841 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.840+0000 7fd98dffb640 1 -- 172.21.15.66:0/482501852 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 505308833 0 0) 0x7fd998140590 con 0x7fd9980b4d60 2024-03-25T18:12:50.841 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.840+0000 7fd98dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:12:50.841 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.840+0000 7fd98dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:50.841 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.840+0000 7fd98dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:50.841 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.840+0000 7fd98dffb640 10 monclient(hunting): my global_id is 4625 2024-03-25T18:12:50.842 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.840+0000 7fd98dffb640 1 -- 172.21.15.66:0/482501852 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd998141fc0 con 0x7fd9980b4d60 2024-03-25T18:12:50.842 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.841+0000 7fd98dffb640 1 -- 172.21.15.66:0/482501852 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1319322500 0 0) 0x7fd9981db590 con 0x7fd99814be90 2024-03-25T18:12:50.842 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.841+0000 7fd98dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:12:50.842 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.841+0000 7fd98dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:50.843 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.841+0000 7fd98dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:50.843 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.841+0000 7fd98dffb640 10 monclient(hunting): my global_id is 4629 2024-03-25T18:12:50.843 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.841+0000 7fd98dffb640 1 -- 172.21.15.66:0/482501852 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd998140590 con 0x7fd99814be90 2024-03-25T18:12:50.854 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.852+0000 7fd98dffb640 1 -- 172.21.15.66:0/482501852 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2261768564 0 0) 0x7fd998140590 con 0x7fd99814be90 2024-03-25T18:12:50.854 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.852+0000 7fd98dffb640 1 -- 172.21.15.66:0/482501852 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9981db590 con 0x7fd99814be90 2024-03-25T18:12:50.854 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.852+0000 7fd98dffb640 1 -- 172.21.15.66:0/482501852 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fd990001bd0 con 0x7fd99814be90 2024-03-25T18:12:50.854 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.852+0000 7fd98dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:50.854 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.852+0000 7fd98dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:12:50.854 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.852+0000 7fd98dffb640 10 monclient(hunting): dump: 2024-03-25T18:12:50.854 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:50.854 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:50.854 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:50.855 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:50.855 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:50.855 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:50.855 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:50.855 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:50.855 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:50.855 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:50.855 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.852+0000 7fd98dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:50.858 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.857+0000 7fd98dffb640 1 -- 172.21.15.66:0/482501852 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2578611027 0 0) 0x7fd97c004eb0 con 0x7fd9980b1730 2024-03-25T18:12:50.858 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.857+0000 7fd98dffb640 1 -- 172.21.15.66:0/482501852 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd998140590 con 0x7fd9980b1730 2024-03-25T18:12:50.859 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.857+0000 7fd98dffb640 1 -- 172.21.15.66:0/482501852 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fd994001a10 con 0x7fd9980b1730 2024-03-25T18:12:50.859 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.857+0000 7fd98dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:50.859 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.857+0000 7fd98dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:50.859 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.857+0000 7fd98dffb640 10 monclient(hunting): dump: 2024-03-25T18:12:50.859 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:50.859 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:50.859 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:50.859 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:50.859 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:50.859 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:50.859 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:50.859 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:50.859 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:50.859 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:50.860 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.858+0000 7fd98dffb640 1 -- 172.21.15.66:0/482501852 <== mon.0 v1:172.21.15.66:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1329851378 0 0) 0x7fd9981db590 con 0x7fd99814be90 2024-03-25T18:12:50.860 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.858+0000 7fd98dffb640 1 -- 172.21.15.66:0/482501852 >> v1:172.21.15.66:6790/0 conn(0x7fd9980b4d60 legacy=0x7fd998134e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:50.860 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.858+0000 7fd98dffb640 1 -- 172.21.15.66:0/482501852 >> v1:172.21.15.193:6789/0 conn(0x7fd9980b1730 legacy=0x7fd9981dae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:50.861 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.858+0000 7fd98dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:12:50.861 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.858+0000 7fd98dffb640 1 monclient: found mon.a 2024-03-25T18:12:50.861 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.858+0000 7fd98dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:50.861 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.858+0000 7fd98dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:12:50.861 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.858+0000 7fd98dffb640 1 -- 172.21.15.66:0/482501852 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9981d5440 con 0x7fd99814be90 2024-03-25T18:12:50.861 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.858+0000 7fd98dffb640 10 monclient: _finish_auth 0 2024-03-25T18:12:50.861 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.858+0000 7fd98dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:12:50.861 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.858+0000 7fd98dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:50.861 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.858+0000 7fd99f6e4640 5 monclient: authenticate success, global_id 4629 2024-03-25T18:12:50.861 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.858+0000 7fd99f6e4640 10 monclient: _renew_subs 2024-03-25T18:12:50.861 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.858+0000 7fd99f6e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:12:50.861 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.858+0000 7fd99f6e4640 1 -- 172.21.15.66:0/482501852 --> v1:172.21.15.66:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd998135550 con 0x7fd99814be90 2024-03-25T18:12:50.861 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.858+0000 7fd99f6e4640 10 client.4629.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:50.861 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.858+0000 7fd99f6e4640 10 monclient: _renew_subs 2024-03-25T18:12:50.861 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.858+0000 7fd99f6e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:12:50.861 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.858+0000 7fd99f6e4640 1 -- 172.21.15.66:0/482501852 --> v1:172.21.15.66:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd998137c40 con 0x7fd99814be90 2024-03-25T18:12:50.862 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.858+0000 7fd99f6e4640 1 librados: init done 2024-03-25T18:12:50.862 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.859+0000 7fd99f6e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:50.862 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.859+0000 7fd99f6e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:50.862 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.859+0000 7fd99f6e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:12:50.862 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:50.859+0000 7fd99f6e4640 1 -- 172.21.15.66:0/482501852 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd998135850 con 0x7fd99814be90 2024-03-25T18:12:51.176 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.174+0000 7fd98dffb640 1 -- 172.21.15.66:0/482501852 <== mon.0 v1:172.21.15.66:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd990002020 con 0x7fd99814be90 2024-03-25T18:12:51.176 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.174+0000 7fd98dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:51.176 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.174+0000 7fd98dffb640 1 -- 172.21.15.66:0/482501852 <== mon.0 v1:172.21.15.66:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fd990004d00 con 0x7fd99814be90 2024-03-25T18:12:51.176 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.174+0000 7fd98dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:51.176 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.174+0000 7fd98dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:51.176 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.174+0000 7fd98dffb640 10 monclient: dump: 2024-03-25T18:12:51.176 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:51.177 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:51.177 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:51.177 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:51.177 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:51.177 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:51.177 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:51.177 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:51.177 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:51.177 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:51.177 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.174+0000 7fd98f7fe640 4 set_mon_vals no callback set 2024-03-25T18:12:51.178 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.175+0000 7fd98dffb640 1 -- 172.21.15.66:0/482501852 <== mon.0 v1:172.21.15.66:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7fd99001ad10 con 0x7fd99814be90 2024-03-25T18:12:51.178 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.176+0000 7fd99d480640 10 monclient: get_auth_request con 0x7fd97c06df70 auth_method 0 2024-03-25T18:12:51.178 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.176+0000 7fd98dffb640 10 client.4629.objecter ms_handle_connect 0x7fd97c06df70 2024-03-25T18:12:51.183 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.182+0000 7fd98dffb640 1 -- 172.21.15.66:0/482501852 <== mon.0 v1:172.21.15.66:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7fd990054900 con 0x7fd99814be90 2024-03-25T18:12:51.183 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.182+0000 7fd98dffb640 10 client.4629.objecter ms_dispatch 0x7fd998000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:51.184 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.182+0000 7fd98dffb640 3 client.4629.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:51.184 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.182+0000 7fd98dffb640 3 client.4629.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:51.184 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.182+0000 7fd98dffb640 20 client.4629.objecter dump_active .. 0 homeless 2024-03-25T18:12:51.184 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.182+0000 7fd98dffb640 1 -- 172.21.15.66:0/482501852 <== mon.0 v1:172.21.15.66:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd990050240 con 0x7fd99814be90 2024-03-25T18:12:51.184 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.182+0000 7fd98dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:51.184 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.182+0000 7fd98dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:51.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.403+0000 7fd99f6e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:51.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.403+0000 7fd99f6e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:51.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.403+0000 7fd99f6e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:12:51.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.403+0000 7fd99f6e4640 1 -- 172.21.15.66:0/482501852 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd998140720 con 0x7fd99814be90 2024-03-25T18:12:51.424 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.423+0000 7fd98dffb640 1 -- 172.21.15.66:0/482501852 <== mon.0 v1:172.21.15.66:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7fd990052340 con 0x7fd99814be90 2024-03-25T18:12:51.424 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.423+0000 7fd98dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:51.424 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.423+0000 7fd98dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:51.425 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:12:51.425 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:51.427 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.425+0000 7fd99f6e4640 10 librados: watch_flush enter 2024-03-25T18:12:51.427 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.425+0000 7fd99f6e4640 10 librados: watch_flush exit 2024-03-25T18:12:51.427 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.425+0000 7fd99f6e4640 20 client.4629.objecter shutdown clearing up homeless session... 2024-03-25T18:12:51.427 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.425+0000 7fd99f6e4640 10 client.4629.objecter successfully canceled tick 2024-03-25T18:12:51.427 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.425+0000 7fd99f6e4640 1 -- 172.21.15.66:0/482501852 >> v1:172.21.15.193:6800/2275676803 conn(0x7fd97c06df70 legacy=0x7fd97c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:51.427 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.425+0000 7fd99f6e4640 10 monclient: shutdown 2024-03-25T18:12:51.428 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.425+0000 7fd99f6e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:51.428 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.425+0000 7fd99f6e4640 1 -- 172.21.15.66:0/482501852 >> v1:172.21.15.66:6789/0 conn(0x7fd99814be90 legacy=0x7fd9981e3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:51.428 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.425+0000 7fd99f6e4640 1 -- 172.21.15.66:0/482501852 shutdown_connections 2024-03-25T18:12:51.428 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.425+0000 7fd99f6e4640 1 -- 172.21.15.66:0/482501852 >> 172.21.15.66:0/482501852 conn(0x7fd9980617a0 msgr2=0x7fd99813c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:51.428 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.425+0000 7fd99f6e4640 1 -- 172.21.15.66:0/482501852 shutdown_connections 2024-03-25T18:12:51.428 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.425+0000 7fd99f6e4640 1 -- 172.21.15.66:0/482501852 wait complete. 2024-03-25T18:12:51.428 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:51.425+0000 7fd99f6e4640 1 librados: shutdown 2024-03-25T18:12:51.438 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:58.245 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:58.366 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.364+0000 7f7c5a2c1640 10 monclient: get_monmap_and_config 2024-03-25T18:12:58.366 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.364+0000 7f7c5a2c1640 10 monclient: build_initial_monmap 2024-03-25T18:12:58.366 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.364+0000 7f7c5a2c1640 10 monclient: monmap: 2024-03-25T18:12:58.366 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:12:58.367 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:58.367 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:12:58.366444+0000 2024-03-25T18:12:58.367 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:12:58.366444+0000 2024-03-25T18:12:58.367 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:58.367 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:58.367 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:12:58.367 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:12:58.367 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:12:58.367 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:58.368 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.365+0000 7f7c5a2c1640 1 Processor -- start 2024-03-25T18:12:58.369 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.365+0000 7f7c5a2c1640 1 -- start start 2024-03-25T18:12:58.369 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.365+0000 7f7c5a2c1640 10 monclient: init 2024-03-25T18:12:58.369 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.366+0000 7f7c5a2c1640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:58.369 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.366+0000 7f7c5a2c1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:12:58.369 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.366+0000 7f7c5a2c1640 10 monclient(hunting): picked mon.noname-a con 0x7f7c540b4d60 addr v1:172.21.15.66:6789/0 2024-03-25T18:12:58.369 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.366+0000 7f7c5a2c1640 10 monclient(hunting): picked mon.noname-b con 0x7f7c540b1730 addr v1:172.21.15.66:6790/0 2024-03-25T18:12:58.369 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.367+0000 7f7c5a2c1640 10 monclient(hunting): picked mon.noname-c con 0x7f7c5414be90 addr v1:172.21.15.193:6789/0 2024-03-25T18:12:58.369 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.367+0000 7f7c5a2c1640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c540af940 con 0x7f7c540b4d60 2024-03-25T18:12:58.369 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.367+0000 7f7c5a2c1640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c540b0a70 con 0x7f7c5414be90 2024-03-25T18:12:58.369 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.367+0000 7f7c5a2c1640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c540b0d50 con 0x7f7c540b1730 2024-03-25T18:12:58.369 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.367+0000 7f7c5a2c1640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:58.369 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.367+0000 7f7c5a2c1640 10 monclient(hunting): authenticate will time out at 7527.427171s 2024-03-25T18:12:58.380 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.379+0000 7f7c5885e640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f7c5414be90 0x7f7c5414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:43724/0 (socket says 172.21.15.66:43724) 2024-03-25T18:12:58.380 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.379+0000 7f7c5885e640 1 -- 172.21.15.66:0/3621455209 learned_addr learned my addr 172.21.15.66:0/3621455209 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:12:58.389 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.388+0000 7f7c52ffd640 1 -- 172.21.15.66:0/3621455209 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1703416311 0 0) 0x7f7c540b0a70 con 0x7f7c5414be90 2024-03-25T18:12:58.389 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.388+0000 7f7c52ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:58.389 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.388+0000 7f7c52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:58.389 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.388+0000 7f7c52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:58.389 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.388+0000 7f7c52ffd640 10 monclient(hunting): my global_id is 4627 2024-03-25T18:12:58.390 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.388+0000 7f7c52ffd640 1 -- 172.21.15.66:0/3621455209 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c3c003520 con 0x7f7c5414be90 2024-03-25T18:12:58.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.405+0000 7f7c52ffd640 1 -- 172.21.15.66:0/3621455209 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1424628155 0 0) 0x7f7c540b0d50 con 0x7f7c540b1730 2024-03-25T18:12:58.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.405+0000 7f7c52ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:58.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.405+0000 7f7c52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:58.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.405+0000 7f7c52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:58.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.405+0000 7f7c52ffd640 10 monclient(hunting): my global_id is 4628 2024-03-25T18:12:58.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.405+0000 7f7c52ffd640 1 -- 172.21.15.66:0/3621455209 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c540b0a70 con 0x7f7c540b1730 2024-03-25T18:12:58.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.405+0000 7f7c52ffd640 1 -- 172.21.15.66:0/3621455209 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1219553731 0 0) 0x7f7c540af940 con 0x7f7c540b4d60 2024-03-25T18:12:58.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.405+0000 7f7c52ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:58.408 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.405+0000 7f7c52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:58.408 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.405+0000 7f7c52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:58.408 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.405+0000 7f7c52ffd640 10 monclient(hunting): my global_id is 4632 2024-03-25T18:12:58.408 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.405+0000 7f7c52ffd640 1 -- 172.21.15.66:0/3621455209 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c540b0d50 con 0x7f7c540b4d60 2024-03-25T18:12:58.412 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.410+0000 7f7c52ffd640 1 -- 172.21.15.66:0/3621455209 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 971010699 0 0) 0x7f7c3c003520 con 0x7f7c5414be90 2024-03-25T18:12:58.412 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.410+0000 7f7c52ffd640 1 -- 172.21.15.66:0/3621455209 >> v1:172.21.15.66:6790/0 conn(0x7f7c540b1730 legacy=0x7f7c540b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:58.412 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.410+0000 7f7c52ffd640 1 -- 172.21.15.66:0/3621455209 >> v1:172.21.15.66:6789/0 conn(0x7f7c540b4d60 legacy=0x7f7c5413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:58.412 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.410+0000 7f7c52ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:12:58.412 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.410+0000 7f7c52ffd640 1 monclient: found mon.noname-c 2024-03-25T18:12:58.412 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.410+0000 7f7c52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:58.412 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.410+0000 7f7c52ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:12:58.412 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.410+0000 7f7c52ffd640 1 -- 172.21.15.66:0/3621455209 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c540b1030 con 0x7f7c5414be90 2024-03-25T18:12:58.412 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.410+0000 7f7c52ffd640 10 monclient: _finish_auth 0 2024-03-25T18:12:58.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.410+0000 7f7c52ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:12:58.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.410+0000 7f7c52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:58.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.410+0000 7f7c52ffd640 1 -- 172.21.15.66:0/3621455209 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7c4c001a10 con 0x7f7c5414be90 2024-03-25T18:12:58.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.410+0000 7f7c52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:58.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.410+0000 7f7c52ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:58.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.410+0000 7f7c52ffd640 10 monclient: dump: 2024-03-25T18:12:58.413 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:58.413 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:58.413 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:58.413 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:58.413 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:58.413 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:58.413 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:58.413 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:58.413 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:58.414 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:58.414 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.410+0000 7f7c5a2c1640 5 monclient: authenticate success, global_id 4627 2024-03-25T18:12:58.414 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.410+0000 7f7c5a2c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:58.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.435+0000 7f7c52ffd640 1 -- 172.21.15.66:0/3621455209 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c4c001b70 con 0x7f7c5414be90 2024-03-25T18:12:58.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.435+0000 7f7c52ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:58.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.435+0000 7f7c52ffd640 1 -- 172.21.15.66:0/3621455209 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7c4c001f80 con 0x7f7c5414be90 2024-03-25T18:12:58.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.435+0000 7f7c52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:58.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.435+0000 7f7c52ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:58.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.435+0000 7f7c52ffd640 10 monclient: dump: 2024-03-25T18:12:58.437 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:58.437 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:58.437 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:58.437 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:58.437 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:58.437 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:58.437 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:58.438 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:58.438 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:58.438 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:58.438 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.435+0000 7f7c5a2c1640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:58.438 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.435+0000 7f7c5a2c1640 4 set_mon_vals no callback set 2024-03-25T18:12:58.439 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.436+0000 7f7c5a2c1640 10 monclient: shutdown 2024-03-25T18:12:58.439 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.436+0000 7f7c5a2c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:58.439 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.436+0000 7f7c5a2c1640 1 -- 172.21.15.66:0/3621455209 >> v1:172.21.15.193:6789/0 conn(0x7f7c5414be90 legacy=0x7f7c5414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:58.440 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.436+0000 7f7c5a2c1640 1 -- 172.21.15.66:0/3621455209 shutdown_connections 2024-03-25T18:12:58.440 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.436+0000 7f7c5a2c1640 1 -- 172.21.15.66:0/3621455209 >> 172.21.15.66:0/3621455209 conn(0x7f7c540617a0 msgr2=0x7f7c54061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:58.440 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.436+0000 7f7c5a2c1640 1 -- 172.21.15.66:0/3621455209 shutdown_connections 2024-03-25T18:12:58.440 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.436+0000 7f7c5a2c1640 1 -- 172.21.15.66:0/3621455209 wait complete. 2024-03-25T18:12:58.440 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.437+0000 7f7c5a2c1640 10 monclient: build_initial_monmap 2024-03-25T18:12:58.440 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.437+0000 7f7c5a2c1640 10 monclient: monmap: 2024-03-25T18:12:58.440 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:12:58.440 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:58.440 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:12:58.440 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:12:58.440 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:58.440 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:12:58.440 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:12:58.440 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:12:58.441 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:12:58.441 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:58.441 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.437+0000 7f7c5a2c1640 1 librados: starting msgr at 2024-03-25T18:12:58.441 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.437+0000 7f7c5a2c1640 1 librados: starting objecter 2024-03-25T18:12:58.441 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.437+0000 7f7c5a2c1640 1 Processor -- start 2024-03-25T18:12:58.441 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.438+0000 7f7c5a2c1640 1 -- start start 2024-03-25T18:12:58.441 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.438+0000 7f7c5a2c1640 1 librados: setting wanted keys 2024-03-25T18:12:58.441 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.438+0000 7f7c5a2c1640 1 librados: calling monclient init 2024-03-25T18:12:58.441 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.438+0000 7f7c5a2c1640 10 monclient: init 2024-03-25T18:12:58.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.439+0000 7f7c5a2c1640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:58.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.439+0000 7f7c5a2c1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:12:58.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.439+0000 7f7c5a2c1640 10 monclient(hunting): picked mon.noname-a con 0x7f7c540b1730 addr v1:172.21.15.66:6789/0 2024-03-25T18:12:58.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.439+0000 7f7c5a2c1640 10 monclient(hunting): picked mon.noname-b con 0x7f7c540b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T18:12:58.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.439+0000 7f7c5a2c1640 10 monclient(hunting): picked mon.noname-c con 0x7f7c5414be90 addr v1:172.21.15.66:6790/0 2024-03-25T18:12:58.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.439+0000 7f7c5a2c1640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c541363f0 con 0x7f7c540b1730 2024-03-25T18:12:58.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.439+0000 7f7c5a2c1640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c541374e0 con 0x7f7c540b4d60 2024-03-25T18:12:58.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.439+0000 7f7c5a2c1640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c54138140 con 0x7f7c5414be90 2024-03-25T18:12:58.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.439+0000 7f7c5a2c1640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:58.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.439+0000 7f7c5a2c1640 10 monclient(hunting): authenticate will time out at 7527.499712s 2024-03-25T18:12:58.449 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.447+0000 7f7c53fff640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f7c540b1730 0x7f7c54136dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:35728/0 (socket says 172.21.15.66:35728) 2024-03-25T18:12:58.449 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.447+0000 7f7c53fff640 1 -- 172.21.15.66:0/3650008928 learned_addr learned my addr 172.21.15.66:0/3650008928 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:12:58.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.449+0000 7f7c50ff9640 10 client.?.objecter ms_handle_connect 0x7f7c540b1730 2024-03-25T18:12:58.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.449+0000 7f7c50ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:58.456 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.454+0000 7f7c50ff9640 10 client.?.objecter ms_handle_connect 0x7f7c540b4d60 2024-03-25T18:12:58.456 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.454+0000 7f7c50ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:58.458 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.457+0000 7f7c50ff9640 10 client.?.objecter ms_handle_connect 0x7f7c5414be90 2024-03-25T18:12:58.458 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.457+0000 7f7c50ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:58.462 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.461+0000 7f7c50ff9640 1 -- 172.21.15.66:0/3650008928 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3971913159 0 0) 0x7f7c54138140 con 0x7f7c5414be90 2024-03-25T18:12:58.462 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.461+0000 7f7c50ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:58.462 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.461+0000 7f7c50ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:58.463 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.461+0000 7f7c50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:58.463 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.461+0000 7f7c50ff9640 10 monclient(hunting): my global_id is 4631 2024-03-25T18:12:58.463 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.461+0000 7f7c50ff9640 1 -- 172.21.15.66:0/3650008928 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c40004eb0 con 0x7f7c5414be90 2024-03-25T18:12:58.468 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.466+0000 7f7c50ff9640 1 -- 172.21.15.66:0/3650008928 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3308147383 0 0) 0x7f7c541363f0 con 0x7f7c540b1730 2024-03-25T18:12:58.468 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.466+0000 7f7c50ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:58.468 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.466+0000 7f7c50ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:58.468 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.466+0000 7f7c50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:58.468 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.466+0000 7f7c50ff9640 10 monclient(hunting): my global_id is 4635 2024-03-25T18:12:58.468 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.466+0000 7f7c50ff9640 1 -- 172.21.15.66:0/3650008928 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c54138140 con 0x7f7c540b1730 2024-03-25T18:12:58.471 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.469+0000 7f7c50ff9640 1 -- 172.21.15.66:0/3650008928 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1751391621 0 0) 0x7f7c541374e0 con 0x7f7c540b4d60 2024-03-25T18:12:58.471 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.469+0000 7f7c50ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:58.471 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.469+0000 7f7c50ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:58.471 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.469+0000 7f7c50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:58.471 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.469+0000 7f7c50ff9640 10 monclient(hunting): my global_id is 4630 2024-03-25T18:12:58.471 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.469+0000 7f7c50ff9640 1 -- 172.21.15.66:0/3650008928 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c541363f0 con 0x7f7c540b4d60 2024-03-25T18:12:58.484 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.482+0000 7f7c50ff9640 1 -- 172.21.15.66:0/3650008928 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2051654791 0 0) 0x7f7c541363f0 con 0x7f7c540b4d60 2024-03-25T18:12:58.484 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.482+0000 7f7c50ff9640 1 -- 172.21.15.66:0/3650008928 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c541374e0 con 0x7f7c540b4d60 2024-03-25T18:12:58.484 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.482+0000 7f7c50ff9640 1 -- 172.21.15.66:0/3650008928 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7c48001930 con 0x7f7c540b4d60 2024-03-25T18:12:58.484 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.482+0000 7f7c50ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:58.484 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.482+0000 7f7c50ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:58.484 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.482+0000 7f7c50ff9640 10 monclient(hunting): dump: 2024-03-25T18:12:58.484 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:58.484 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:58.484 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:58.484 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:58.485 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:58.485 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:58.485 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:58.485 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:58.485 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:58.485 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:58.485 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.482+0000 7f7c50ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:58.490 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.488+0000 7f7c50ff9640 1 -- 172.21.15.66:0/3650008928 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2048012937 0 0) 0x7f7c541374e0 con 0x7f7c540b4d60 2024-03-25T18:12:58.490 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.488+0000 7f7c50ff9640 1 -- 172.21.15.66:0/3650008928 >> v1:172.21.15.66:6790/0 conn(0x7f7c5414be90 legacy=0x7f7c54143c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:58.490 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.488+0000 7f7c50ff9640 1 -- 172.21.15.66:0/3650008928 >> v1:172.21.15.66:6789/0 conn(0x7f7c540b1730 legacy=0x7f7c54136dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:58.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.488+0000 7f7c50ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:12:58.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.488+0000 7f7c50ff9640 1 monclient: found mon.b 2024-03-25T18:12:58.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.488+0000 7f7c50ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:58.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.488+0000 7f7c50ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:12:58.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.488+0000 7f7c50ff9640 1 -- 172.21.15.66:0/3650008928 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c54139090 con 0x7f7c540b4d60 2024-03-25T18:12:58.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.488+0000 7f7c50ff9640 10 monclient: _finish_auth 0 2024-03-25T18:12:58.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.488+0000 7f7c50ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:12:58.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.488+0000 7f7c50ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:58.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.488+0000 7f7c5a2c1640 5 monclient: authenticate success, global_id 4630 2024-03-25T18:12:58.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.488+0000 7f7c5a2c1640 10 monclient: _renew_subs 2024-03-25T18:12:58.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.488+0000 7f7c5a2c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:12:58.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.488+0000 7f7c5a2c1640 1 -- 172.21.15.66:0/3650008928 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c54136030 con 0x7f7c540b4d60 2024-03-25T18:12:58.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.488+0000 7f7c5a2c1640 10 client.4630.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:58.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.488+0000 7f7c5a2c1640 10 monclient: _renew_subs 2024-03-25T18:12:58.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.488+0000 7f7c5a2c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:12:58.491 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.488+0000 7f7c5a2c1640 1 -- 172.21.15.66:0/3650008928 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c54130650 con 0x7f7c540b4d60 2024-03-25T18:12:58.492 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.488+0000 7f7c5a2c1640 1 librados: init done 2024-03-25T18:12:58.492 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.489+0000 7f7c5a2c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:58.492 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.489+0000 7f7c5a2c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:58.492 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.489+0000 7f7c5a2c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:12:58.492 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.489+0000 7f7c5a2c1640 1 -- 172.21.15.66:0/3650008928 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c540ae6c0 con 0x7f7c540b4d60 2024-03-25T18:12:58.770 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.769+0000 7f7c50ff9640 1 -- 172.21.15.66:0/3650008928 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c480035f0 con 0x7f7c540b4d60 2024-03-25T18:12:58.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.769+0000 7f7c50ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:58.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.769+0000 7f7c50ff9640 1 -- 172.21.15.66:0/3650008928 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7c48003900 con 0x7f7c540b4d60 2024-03-25T18:12:58.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.769+0000 7f7c50ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:58.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.769+0000 7f7c50ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:58.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.769+0000 7f7c50ff9640 10 monclient: dump: 2024-03-25T18:12:58.771 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:12:58.771 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:12:58.771 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:58.771 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:12:58.771 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:58.771 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:12:58.771 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:12:58.772 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:12:58.772 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:12:58.772 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:12:58.772 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.769+0000 7f7c527fc640 4 set_mon_vals no callback set 2024-03-25T18:12:58.772 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.770+0000 7f7c50ff9640 1 -- 172.21.15.66:0/3650008928 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f7c480198d0 con 0x7f7c540b4d60 2024-03-25T18:12:58.773 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.770+0000 7f7c53fff640 10 monclient: get_auth_request con 0x7f7c4006de20 auth_method 0 2024-03-25T18:12:58.773 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.771+0000 7f7c50ff9640 10 client.4630.objecter ms_handle_connect 0x7f7c4006de20 2024-03-25T18:12:58.776 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.775+0000 7f7c50ff9640 1 -- 172.21.15.66:0/3650008928 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f7c48053580 con 0x7f7c540b4d60 2024-03-25T18:12:58.776 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.775+0000 7f7c50ff9640 10 client.4630.objecter ms_dispatch 0x7f7c54000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:58.776 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.775+0000 7f7c50ff9640 3 client.4630.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:58.777 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.775+0000 7f7c50ff9640 3 client.4630.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:58.777 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.775+0000 7f7c50ff9640 20 client.4630.objecter dump_active .. 0 homeless 2024-03-25T18:12:58.777 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.776+0000 7f7c50ff9640 1 -- 172.21.15.66:0/3650008928 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7c4804eec0 con 0x7f7c540b4d60 2024-03-25T18:12:58.777 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.776+0000 7f7c50ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:58.777 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.776+0000 7f7c50ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:58.996 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.994+0000 7f7c5a2c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:58.996 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.994+0000 7f7c5a2c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:58.996 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.994+0000 7f7c5a2c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:12:58.996 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:58.994+0000 7f7c5a2c1640 1 -- 172.21.15.66:0/3650008928 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c540627b0 con 0x7f7c540b4d60 2024-03-25T18:12:59.083 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:59.081+0000 7f7c50ff9640 1 -- 172.21.15.66:0/3650008928 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f7c48050fc0 con 0x7f7c540b4d60 2024-03-25T18:12:59.083 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:59.081+0000 7f7c50ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:59.083 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:59.081+0000 7f7c50ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:59.083 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:12:59.083 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:59.085 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:59.083+0000 7f7c5a2c1640 10 librados: watch_flush enter 2024-03-25T18:12:59.085 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:59.083+0000 7f7c5a2c1640 10 librados: watch_flush exit 2024-03-25T18:12:59.086 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:59.083+0000 7f7c5a2c1640 20 client.4630.objecter shutdown clearing up homeless session... 2024-03-25T18:12:59.086 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:59.083+0000 7f7c5a2c1640 10 client.4630.objecter successfully canceled tick 2024-03-25T18:12:59.086 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:59.083+0000 7f7c5a2c1640 1 -- 172.21.15.66:0/3650008928 >> v1:172.21.15.193:6800/2275676803 conn(0x7f7c4006de20 legacy=0x7f7c400702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:59.086 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:59.083+0000 7f7c5a2c1640 10 monclient: shutdown 2024-03-25T18:12:59.086 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:59.083+0000 7f7c5a2c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:59.086 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:59.083+0000 7f7c5a2c1640 1 -- 172.21.15.66:0/3650008928 >> v1:172.21.15.193:6789/0 conn(0x7f7c540b4d60 legacy=0x7f7c5412fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:59.086 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:59.083+0000 7f7c5a2c1640 1 -- 172.21.15.66:0/3650008928 shutdown_connections 2024-03-25T18:12:59.086 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:59.083+0000 7f7c5a2c1640 1 -- 172.21.15.66:0/3650008928 >> 172.21.15.66:0/3650008928 conn(0x7f7c540617a0 msgr2=0x7f7c5413c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:59.086 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:59.084+0000 7f7c5a2c1640 1 -- 172.21.15.66:0/3650008928 shutdown_connections 2024-03-25T18:12:59.086 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:59.084+0000 7f7c5a2c1640 1 -- 172.21.15.66:0/3650008928 wait complete. 2024-03-25T18:12:59.086 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:12:59.084+0000 7f7c5a2c1640 1 librados: shutdown 2024-03-25T18:12:59.098 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:05.904 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:06.025 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.024+0000 7f2fcfcb3640 10 monclient: get_monmap_and_config 2024-03-25T18:13:06.025 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.024+0000 7f2fcfcb3640 10 monclient: build_initial_monmap 2024-03-25T18:13:06.026 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.024+0000 7f2fcfcb3640 10 monclient: monmap: 2024-03-25T18:13:06.026 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:13:06.026 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:06.026 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:13:06.025575+0000 2024-03-25T18:13:06.026 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:13:06.025575+0000 2024-03-25T18:13:06.026 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:06.026 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:06.026 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:13:06.026 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:13:06.026 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:13:06.026 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:06.027 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.024+0000 7f2fcfcb3640 1 Processor -- start 2024-03-25T18:13:06.028 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.025+0000 7f2fcfcb3640 1 -- start start 2024-03-25T18:13:06.028 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.025+0000 7f2fcfcb3640 10 monclient: init 2024-03-25T18:13:06.028 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.025+0000 7f2fcfcb3640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:06.028 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.025+0000 7f2fcfcb3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:13:06.028 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.025+0000 7f2fcfcb3640 10 monclient(hunting): picked mon.noname-b con 0x7f2fc80a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:13:06.028 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.025+0000 7f2fcfcb3640 10 monclient(hunting): picked mon.noname-a con 0x7f2fc813f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:13:06.028 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.025+0000 7f2fcfcb3640 10 monclient(hunting): picked mon.noname-c con 0x7f2fc81430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:13:06.028 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.025+0000 7f2fcfcb3640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc8142520 con 0x7f2fc813f710 2024-03-25T18:13:06.028 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.025+0000 7f2fcfcb3640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc80b28a0 con 0x7f2fc81430c0 2024-03-25T18:13:06.028 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.026+0000 7f2fcfcb3640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc80b2a30 con 0x7f2fc80a98f0 2024-03-25T18:13:06.028 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.026+0000 7f2fcfcb3640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:06.028 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.026+0000 7f2fcfcb3640 10 monclient(hunting): authenticate will time out at 7535.086126s 2024-03-25T18:13:06.039 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.037+0000 7f2fcda4f640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f2fc80a98f0 0x7f2fc80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:53874/0 (socket says 172.21.15.66:53874) 2024-03-25T18:13:06.039 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.037+0000 7f2fcda4f640 1 -- 172.21.15.66:0/2580868863 learned_addr learned my addr 172.21.15.66:0/2580868863 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:13:06.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.050+0000 7f2fcca4d640 1 -- 172.21.15.66:0/2580868863 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177664120 0 0) 0x7f2fc80b2a30 con 0x7f2fc80a98f0 2024-03-25T18:13:06.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.050+0000 7f2fcca4d640 10 monclient(hunting): _negotiate 2024-03-25T18:13:06.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.050+0000 7f2fcca4d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:06.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.050+0000 7f2fcca4d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:06.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.050+0000 7f2fcca4d640 10 monclient(hunting): my global_id is 4634 2024-03-25T18:13:06.052 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.050+0000 7f2fcca4d640 1 -- 172.21.15.66:0/2580868863 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb4003520 con 0x7f2fc80a98f0 2024-03-25T18:13:06.056 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.054+0000 7f2fcca4d640 1 -- 172.21.15.66:0/2580868863 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3902775547 0 0) 0x7f2fc80b28a0 con 0x7f2fc81430c0 2024-03-25T18:13:06.056 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.054+0000 7f2fcca4d640 10 monclient(hunting): _negotiate 2024-03-25T18:13:06.056 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.054+0000 7f2fcca4d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:06.056 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.054+0000 7f2fcca4d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:06.056 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.054+0000 7f2fcca4d640 10 monclient(hunting): my global_id is 4633 2024-03-25T18:13:06.056 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.054+0000 7f2fcca4d640 1 -- 172.21.15.66:0/2580868863 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fc80b2a30 con 0x7f2fc81430c0 2024-03-25T18:13:06.063 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.062+0000 7f2fcca4d640 1 -- 172.21.15.66:0/2580868863 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4203515327 0 0) 0x7f2fc80b2a30 con 0x7f2fc81430c0 2024-03-25T18:13:06.063 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.062+0000 7f2fcca4d640 1 -- 172.21.15.66:0/2580868863 >> v1:172.21.15.66:6790/0 conn(0x7f2fc80a98f0 legacy=0x7f2fc80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:06.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.062+0000 7f2fcca4d640 1 -- 172.21.15.66:0/2580868863 >> v1:172.21.15.66:6789/0 conn(0x7f2fc813f710 legacy=0x7f2fc81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:06.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.062+0000 7f2fcca4d640 10 monclient: _finish_hunting 0 2024-03-25T18:13:06.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.062+0000 7f2fcca4d640 1 monclient: found mon.noname-c 2024-03-25T18:13:06.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.062+0000 7f2fcca4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:06.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.062+0000 7f2fcca4d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:13:06.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.062+0000 7f2fcca4d640 1 -- 172.21.15.66:0/2580868863 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc80b44c0 con 0x7f2fc81430c0 2024-03-25T18:13:06.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.062+0000 7f2fcca4d640 10 monclient: _finish_auth 0 2024-03-25T18:13:06.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.062+0000 7f2fcca4d640 10 monclient: _check_auth_tickets 2024-03-25T18:13:06.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.062+0000 7f2fcca4d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:06.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.062+0000 7f2fcca4d640 1 -- 172.21.15.66:0/2580868863 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f2fc00019f0 con 0x7f2fc81430c0 2024-03-25T18:13:06.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.062+0000 7f2fcca4d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:06.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.062+0000 7f2fcca4d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:06.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.062+0000 7f2fcca4d640 10 monclient: dump: 2024-03-25T18:13:06.064 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:06.065 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:06.065 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:06.065 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:06.065 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:06.065 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:06.065 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:06.065 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:06.065 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:06.065 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:06.065 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.062+0000 7f2fcfcb3640 5 monclient: authenticate success, global_id 4633 2024-03-25T18:13:06.065 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.062+0000 7f2fcfcb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:06.075 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.073+0000 7f2fcca4d640 1 -- 172.21.15.66:0/2580868863 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2fc0001b50 con 0x7f2fc81430c0 2024-03-25T18:13:06.075 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.073+0000 7f2fcca4d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:06.075 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.073+0000 7f2fcca4d640 1 -- 172.21.15.66:0/2580868863 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f2fc0001f20 con 0x7f2fc81430c0 2024-03-25T18:13:06.075 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.073+0000 7f2fcfcb3640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:06.075 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.073+0000 7f2fcfcb3640 4 set_mon_vals no callback set 2024-03-25T18:13:06.075 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.073+0000 7f2fcca4d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:06.075 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.073+0000 7f2fcca4d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:06.075 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.073+0000 7f2fcca4d640 10 monclient: dump: 2024-03-25T18:13:06.075 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:06.075 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:06.075 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:06.075 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:06.075 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:06.075 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:06.076 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:06.076 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:06.076 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:06.076 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:06.077 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.074+0000 7f2fcfcb3640 10 monclient: shutdown 2024-03-25T18:13:06.077 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.074+0000 7f2fcfcb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:06.077 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.074+0000 7f2fcfcb3640 1 -- 172.21.15.66:0/2580868863 >> v1:172.21.15.193:6789/0 conn(0x7f2fc81430c0 legacy=0x7f2fc814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:06.077 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.074+0000 7f2fcfcb3640 1 -- 172.21.15.66:0/2580868863 shutdown_connections 2024-03-25T18:13:06.077 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.074+0000 7f2fcfcb3640 1 -- 172.21.15.66:0/2580868863 >> 172.21.15.66:0/2580868863 conn(0x7f2fc80b7ea0 msgr2=0x7f2fc80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:06.077 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.074+0000 7f2fcfcb3640 1 -- 172.21.15.66:0/2580868863 shutdown_connections 2024-03-25T18:13:06.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.074+0000 7f2fcfcb3640 1 -- 172.21.15.66:0/2580868863 wait complete. 2024-03-25T18:13:06.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.075+0000 7f2fcfcb3640 10 monclient: build_initial_monmap 2024-03-25T18:13:06.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.075+0000 7f2fcfcb3640 10 monclient: monmap: 2024-03-25T18:13:06.078 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:13:06.078 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:06.078 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:13:06.078 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:13:06.078 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:06.078 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:13:06.078 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:13:06.078 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:13:06.078 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:13:06.078 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:06.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.075+0000 7f2fcfcb3640 1 librados: starting msgr at 2024-03-25T18:13:06.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.075+0000 7f2fcfcb3640 1 librados: starting objecter 2024-03-25T18:13:06.079 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.075+0000 7f2fcfcb3640 1 Processor -- start 2024-03-25T18:13:06.079 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.075+0000 7f2fcfcb3640 1 -- start start 2024-03-25T18:13:06.079 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.075+0000 7f2fcfcb3640 1 librados: setting wanted keys 2024-03-25T18:13:06.079 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.075+0000 7f2fcfcb3640 1 librados: calling monclient init 2024-03-25T18:13:06.079 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.075+0000 7f2fcfcb3640 10 monclient: init 2024-03-25T18:13:06.080 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.076+0000 7f2fcfcb3640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:06.080 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.076+0000 7f2fcfcb3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:06.080 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.076+0000 7f2fcfcb3640 10 monclient(hunting): picked mon.noname-c con 0x7f2fc80a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:13:06.080 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.076+0000 7f2fcfcb3640 10 monclient(hunting): picked mon.noname-b con 0x7f2fc813f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:13:06.080 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.076+0000 7f2fcfcb3640 10 monclient(hunting): picked mon.noname-a con 0x7f2fc81430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:13:06.080 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.076+0000 7f2fcfcb3640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc806f500 con 0x7f2fc81430c0 2024-03-25T18:13:06.080 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.076+0000 7f2fcfcb3640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc806e710 con 0x7f2fc813f710 2024-03-25T18:13:06.080 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.076+0000 7f2fcfcb3640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc806f730 con 0x7f2fc80a98f0 2024-03-25T18:13:06.080 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.076+0000 7f2fcfcb3640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:06.081 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.076+0000 7f2fcfcb3640 10 monclient(hunting): authenticate will time out at 7535.136972s 2024-03-25T18:13:06.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.087+0000 7f2fcda4f640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f2fc80a98f0 0x7f2fc81db790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:53920/0 (socket says 172.21.15.66:53920) 2024-03-25T18:13:06.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.087+0000 7f2fcda4f640 1 -- 172.21.15.66:0/3627757039 learned_addr learned my addr 172.21.15.66:0/3627757039 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:13:06.092 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.090+0000 7f2fbe7fc640 10 client.?.objecter ms_handle_connect 0x7f2fc80a98f0 2024-03-25T18:13:06.092 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.090+0000 7f2fbe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:06.098 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.096+0000 7f2fbe7fc640 10 client.?.objecter ms_handle_connect 0x7f2fc813f710 2024-03-25T18:13:06.098 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.096+0000 7f2fbe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:06.106 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.104+0000 7f2fbe7fc640 10 client.?.objecter ms_handle_connect 0x7f2fc81430c0 2024-03-25T18:13:06.106 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.104+0000 7f2fbe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:06.117 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.116+0000 7f2fbe7fc640 1 -- 172.21.15.66:0/3627757039 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4157809905 0 0) 0x7f2fc806e710 con 0x7f2fc813f710 2024-03-25T18:13:06.117 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.116+0000 7f2fbe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:06.117 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.116+0000 7f2fbe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:06.117 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.116+0000 7f2fbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:06.117 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.116+0000 7f2fbe7fc640 10 monclient(hunting): my global_id is 4636 2024-03-25T18:13:06.117 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.116+0000 7f2fbe7fc640 1 -- 172.21.15.66:0/3627757039 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fac004eb0 con 0x7f2fc813f710 2024-03-25T18:13:06.121 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.120+0000 7f2fbe7fc640 1 -- 172.21.15.66:0/3627757039 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3451858124 0 0) 0x7f2fc806f500 con 0x7f2fc81430c0 2024-03-25T18:13:06.122 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.120+0000 7f2fbe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:06.122 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.120+0000 7f2fbe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:06.122 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.120+0000 7f2fbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:06.122 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.120+0000 7f2fbe7fc640 10 monclient(hunting): my global_id is 4641 2024-03-25T18:13:06.122 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.120+0000 7f2fbe7fc640 1 -- 172.21.15.66:0/3627757039 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fc806e710 con 0x7f2fc81430c0 2024-03-25T18:13:06.126 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.124+0000 7f2fbe7fc640 1 -- 172.21.15.66:0/3627757039 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1974942005 0 0) 0x7f2fc806f730 con 0x7f2fc80a98f0 2024-03-25T18:13:06.126 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.124+0000 7f2fbe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:06.126 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.124+0000 7f2fbe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:06.126 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.124+0000 7f2fbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:06.126 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.124+0000 7f2fbe7fc640 10 monclient(hunting): my global_id is 4637 2024-03-25T18:13:06.126 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.124+0000 7f2fbe7fc640 1 -- 172.21.15.66:0/3627757039 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fc806f500 con 0x7f2fc80a98f0 2024-03-25T18:13:06.142 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.140+0000 7f2fbe7fc640 1 -- 172.21.15.66:0/3627757039 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3382379520 0 0) 0x7f2fac004eb0 con 0x7f2fc813f710 2024-03-25T18:13:06.142 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.140+0000 7f2fbe7fc640 1 -- 172.21.15.66:0/3627757039 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fc806f730 con 0x7f2fc813f710 2024-03-25T18:13:06.142 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.140+0000 7f2fbe7fc640 1 -- 172.21.15.66:0/3627757039 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f2fb8001930 con 0x7f2fc813f710 2024-03-25T18:13:06.142 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.140+0000 7f2fbe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:06.142 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.140+0000 7f2fbe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:06.142 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.140+0000 7f2fbe7fc640 10 monclient(hunting): dump: 2024-03-25T18:13:06.142 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:06.143 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:06.143 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:06.143 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:06.143 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:06.143 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:06.143 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:06.143 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:06.143 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:06.143 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:06.143 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.140+0000 7f2fbe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:06.146 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.144+0000 7f2fbe7fc640 1 -- 172.21.15.66:0/3627757039 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3331534564 0 0) 0x7f2fc806e710 con 0x7f2fc81430c0 2024-03-25T18:13:06.146 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.145+0000 7f2fbe7fc640 1 -- 172.21.15.66:0/3627757039 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fac004eb0 con 0x7f2fc81430c0 2024-03-25T18:13:06.146 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.145+0000 7f2fbe7fc640 1 -- 172.21.15.66:0/3627757039 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f2fc0001e60 con 0x7f2fc81430c0 2024-03-25T18:13:06.146 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.145+0000 7f2fbe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:06.146 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.145+0000 7f2fbe7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:06.146 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.145+0000 7f2fbe7fc640 10 monclient(hunting): dump: 2024-03-25T18:13:06.146 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:06.147 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:06.147 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:06.147 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:06.147 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:06.147 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:06.147 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:06.147 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:06.147 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:06.147 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:06.153 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.151+0000 7f2fbe7fc640 1 -- 172.21.15.66:0/3627757039 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1579064547 0 0) 0x7f2fc806f730 con 0x7f2fc813f710 2024-03-25T18:13:06.153 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.151+0000 7f2fbe7fc640 1 -- 172.21.15.66:0/3627757039 >> v1:172.21.15.66:6790/0 conn(0x7f2fc80a98f0 legacy=0x7f2fc81db790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:06.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.151+0000 7f2fbe7fc640 1 -- 172.21.15.66:0/3627757039 >> v1:172.21.15.66:6789/0 conn(0x7f2fc81430c0 legacy=0x7f2fc81e70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:06.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.151+0000 7f2fbe7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:13:06.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.151+0000 7f2fbe7fc640 1 monclient: found mon.b 2024-03-25T18:13:06.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.151+0000 7f2fbe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:06.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.151+0000 7f2fbe7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:13:06.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.151+0000 7f2fbe7fc640 1 -- 172.21.15.66:0/3627757039 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc806fa10 con 0x7f2fc813f710 2024-03-25T18:13:06.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.151+0000 7f2fbe7fc640 10 monclient: _finish_auth 0 2024-03-25T18:13:06.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.151+0000 7f2fbe7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:13:06.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.151+0000 7f2fbe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:06.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.151+0000 7f2fcfcb3640 5 monclient: authenticate success, global_id 4636 2024-03-25T18:13:06.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.151+0000 7f2fcfcb3640 10 monclient: _renew_subs 2024-03-25T18:13:06.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.151+0000 7f2fcfcb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:13:06.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.151+0000 7f2fcfcb3640 1 -- 172.21.15.66:0/3627757039 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc814e7d0 con 0x7f2fc813f710 2024-03-25T18:13:06.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.151+0000 7f2fcfcb3640 10 client.4636.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:06.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.152+0000 7f2fcfcb3640 10 monclient: _renew_subs 2024-03-25T18:13:06.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.152+0000 7f2fcfcb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:13:06.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.152+0000 7f2fcfcb3640 1 -- 172.21.15.66:0/3627757039 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc814ec90 con 0x7f2fc813f710 2024-03-25T18:13:06.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.152+0000 7f2fcfcb3640 1 librados: init done 2024-03-25T18:13:06.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.153+0000 7f2fcfcb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:06.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.153+0000 7f2fcfcb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:06.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.153+0000 7f2fcfcb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:13:06.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.153+0000 7f2fcfcb3640 1 -- 172.21.15.66:0/3627757039 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc80627b0 con 0x7f2fc813f710 2024-03-25T18:13:06.427 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.425+0000 7f2fbe7fc640 1 -- 172.21.15.66:0/3627757039 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2fb80035f0 con 0x7f2fc813f710 2024-03-25T18:13:06.427 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.425+0000 7f2fbe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:06.428 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.425+0000 7f2fbe7fc640 1 -- 172.21.15.66:0/3627757039 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f2fb8003980 con 0x7f2fc813f710 2024-03-25T18:13:06.428 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.425+0000 7f2fbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:06.428 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.426+0000 7f2fbe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:06.428 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.426+0000 7f2fbe7fc640 10 monclient: dump: 2024-03-25T18:13:06.428 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:06.428 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:06.428 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:06.428 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:06.428 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:06.428 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:06.428 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:06.428 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:06.428 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:06.428 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:06.429 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.425+0000 7f2fbffff640 4 set_mon_vals no callback set 2024-03-25T18:13:06.429 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.426+0000 7f2fbe7fc640 1 -- 172.21.15.66:0/3627757039 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f2fb8003ba0 con 0x7f2fc813f710 2024-03-25T18:13:06.429 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.427+0000 7f2fcda4f640 10 monclient: get_auth_request con 0x7f2fac06df70 auth_method 0 2024-03-25T18:13:06.429 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.427+0000 7f2fbe7fc640 10 client.4636.objecter ms_handle_connect 0x7f2fac06df70 2024-03-25T18:13:06.436 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.435+0000 7f2fbe7fc640 1 -- 172.21.15.66:0/3627757039 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f2fb8053600 con 0x7f2fc813f710 2024-03-25T18:13:06.436 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.435+0000 7f2fbe7fc640 10 client.4636.objecter ms_dispatch 0x7f2fc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:06.436 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.435+0000 7f2fbe7fc640 3 client.4636.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:06.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.435+0000 7f2fbe7fc640 3 client.4636.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:06.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.435+0000 7f2fbe7fc640 20 client.4636.objecter dump_active .. 0 homeless 2024-03-25T18:13:06.438 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.436+0000 7f2fbe7fc640 1 -- 172.21.15.66:0/3627757039 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2fb804f110 con 0x7f2fc813f710 2024-03-25T18:13:06.438 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.436+0000 7f2fbe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:06.438 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.436+0000 7f2fbe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:06.698 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.697+0000 7f2fcfcb3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:06.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.697+0000 7f2fcfcb3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:06.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.697+0000 7f2fcfcb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:13:06.699 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.697+0000 7f2fcfcb3640 1 -- 172.21.15.66:0/3627757039 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2fc8141a00 con 0x7f2fc813f710 2024-03-25T18:13:06.773 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.771+0000 7f2fbe7fc640 1 -- 172.21.15.66:0/3627757039 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f2fb8051040 con 0x7f2fc813f710 2024-03-25T18:13:06.773 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.771+0000 7f2fbe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:06.773 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.771+0000 7f2fbe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:06.773 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:13:06.773 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:06.775 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.773+0000 7f2fcfcb3640 10 librados: watch_flush enter 2024-03-25T18:13:06.775 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.773+0000 7f2fcfcb3640 10 librados: watch_flush exit 2024-03-25T18:13:06.776 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.773+0000 7f2fcfcb3640 20 client.4636.objecter shutdown clearing up homeless session... 2024-03-25T18:13:06.776 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.773+0000 7f2fcfcb3640 10 client.4636.objecter successfully canceled tick 2024-03-25T18:13:06.776 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.773+0000 7f2fcfcb3640 1 -- 172.21.15.66:0/3627757039 >> v1:172.21.15.193:6800/2275676803 conn(0x7f2fac06df70 legacy=0x7f2fac070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:06.776 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.773+0000 7f2fcfcb3640 10 monclient: shutdown 2024-03-25T18:13:06.776 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.773+0000 7f2fcfcb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:06.776 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.773+0000 7f2fcfcb3640 1 -- 172.21.15.66:0/3627757039 >> v1:172.21.15.193:6789/0 conn(0x7f2fc813f710 legacy=0x7f2fc8071450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:06.776 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.773+0000 7f2fcfcb3640 1 -- 172.21.15.66:0/3627757039 shutdown_connections 2024-03-25T18:13:06.776 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.773+0000 7f2fcfcb3640 1 -- 172.21.15.66:0/3627757039 >> 172.21.15.66:0/3627757039 conn(0x7f2fc80b7ea0 msgr2=0x7f2fc8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:06.776 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.774+0000 7f2fcfcb3640 1 -- 172.21.15.66:0/3627757039 shutdown_connections 2024-03-25T18:13:06.776 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.774+0000 7f2fcfcb3640 1 -- 172.21.15.66:0/3627757039 wait complete. 2024-03-25T18:13:06.776 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:06.774+0000 7f2fcfcb3640 1 librados: shutdown 2024-03-25T18:13:06.786 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:10.314 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:09:41.294 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:10.323 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:09:41.304 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:10.355 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:09:41.335 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:10.374 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:09:41.332 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:10.375 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:09:41.356 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:10.380 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:09:41.339 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:10.386 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:09:41.366 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:10.404 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:09:41.363 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:10.427 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:09:41.408 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:13.593 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:13.703 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.701+0000 7fc3c7d40640 10 monclient: get_monmap_and_config 2024-03-25T18:13:13.703 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.701+0000 7fc3c7d40640 10 monclient: build_initial_monmap 2024-03-25T18:13:13.703 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.701+0000 7fc3c7d40640 10 monclient: monmap: 2024-03-25T18:13:13.703 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:13:13.703 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:13.703 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:13:13.703224+0000 2024-03-25T18:13:13.703 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:13:13.703224+0000 2024-03-25T18:13:13.703 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:13.704 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:13.704 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:13:13.704 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:13:13.704 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:13:13.704 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:13.705 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.702+0000 7fc3c7d40640 1 Processor -- start 2024-03-25T18:13:13.705 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.702+0000 7fc3c7d40640 1 -- start start 2024-03-25T18:13:13.705 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.702+0000 7fc3c7d40640 10 monclient: init 2024-03-25T18:13:13.705 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.703+0000 7fc3c7d40640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:13.705 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.703+0000 7fc3c7d40640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:13:13.705 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.703+0000 7fc3c7d40640 10 monclient(hunting): picked mon.noname-c con 0x7fc3c00b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T18:13:13.706 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.703+0000 7fc3c7d40640 10 monclient(hunting): picked mon.noname-a con 0x7fc3c00b1730 addr v1:172.21.15.66:6789/0 2024-03-25T18:13:13.706 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.703+0000 7fc3c7d40640 10 monclient(hunting): picked mon.noname-b con 0x7fc3c014be90 addr v1:172.21.15.66:6790/0 2024-03-25T18:13:13.706 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.703+0000 7fc3c7d40640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c00af940 con 0x7fc3c00b1730 2024-03-25T18:13:13.706 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.703+0000 7fc3c7d40640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c00b0a70 con 0x7fc3c00b4d60 2024-03-25T18:13:13.706 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.703+0000 7fc3c7d40640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c00b0d50 con 0x7fc3c014be90 2024-03-25T18:13:13.706 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.703+0000 7fc3c7d40640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:13.706 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.703+0000 7fc3c7d40640 10 monclient(hunting): authenticate will time out at 7542.763653s 2024-03-25T18:13:13.717 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.715+0000 7fc3c5adc640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fc3c00b4d60 0x7fc3c013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:46464/0 (socket says 172.21.15.66:46464) 2024-03-25T18:13:13.717 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.715+0000 7fc3c5adc640 1 -- 172.21.15.66:0/3875313361 learned_addr learned my addr 172.21.15.66:0/3875313361 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:13:13.726 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.724+0000 7fc3c4ada640 1 -- 172.21.15.66:0/3875313361 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998352530 0 0) 0x7fc3c00b0a70 con 0x7fc3c00b4d60 2024-03-25T18:13:13.726 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.724+0000 7fc3c4ada640 10 monclient(hunting): _negotiate 2024-03-25T18:13:13.726 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.724+0000 7fc3c4ada640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:13.726 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.724+0000 7fc3c4ada640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:13.726 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.724+0000 7fc3c4ada640 10 monclient(hunting): my global_id is 4639 2024-03-25T18:13:13.726 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.724+0000 7fc3c4ada640 1 -- 172.21.15.66:0/3875313361 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3ac003520 con 0x7fc3c00b4d60 2024-03-25T18:13:13.732 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.730+0000 7fc3c4ada640 1 -- 172.21.15.66:0/3875313361 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 791427734 0 0) 0x7fc3c00b0d50 con 0x7fc3c014be90 2024-03-25T18:13:13.732 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.730+0000 7fc3c4ada640 10 monclient(hunting): _negotiate 2024-03-25T18:13:13.732 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.730+0000 7fc3c4ada640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:13.732 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.730+0000 7fc3c4ada640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:13.732 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.730+0000 7fc3c4ada640 10 monclient(hunting): my global_id is 4640 2024-03-25T18:13:13.732 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.730+0000 7fc3c4ada640 1 -- 172.21.15.66:0/3875313361 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c00b0a70 con 0x7fc3c014be90 2024-03-25T18:13:13.747 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.745+0000 7fc3c4ada640 1 -- 172.21.15.66:0/3875313361 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3765514783 0 0) 0x7fc3ac003520 con 0x7fc3c00b4d60 2024-03-25T18:13:13.747 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.745+0000 7fc3c4ada640 1 -- 172.21.15.66:0/3875313361 >> v1:172.21.15.66:6790/0 conn(0x7fc3c014be90 legacy=0x7fc3c014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:13.747 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.745+0000 7fc3c4ada640 1 -- 172.21.15.66:0/3875313361 >> v1:172.21.15.66:6789/0 conn(0x7fc3c00b1730 legacy=0x7fc3c00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:13.747 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.745+0000 7fc3c4ada640 10 monclient: _finish_hunting 0 2024-03-25T18:13:13.747 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.745+0000 7fc3c4ada640 1 monclient: found mon.noname-c 2024-03-25T18:13:13.747 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.745+0000 7fc3c4ada640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:13.747 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.745+0000 7fc3c4ada640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:13:13.747 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.745+0000 7fc3c4ada640 1 -- 172.21.15.66:0/3875313361 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c00b1030 con 0x7fc3c00b4d60 2024-03-25T18:13:13.747 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.745+0000 7fc3c4ada640 10 monclient: _finish_auth 0 2024-03-25T18:13:13.747 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.745+0000 7fc3c4ada640 10 monclient: _check_auth_tickets 2024-03-25T18:13:13.747 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.745+0000 7fc3c4ada640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:13.748 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.745+0000 7fc3c4ada640 1 -- 172.21.15.66:0/3875313361 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fc3bc001690 con 0x7fc3c00b4d60 2024-03-25T18:13:13.748 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.745+0000 7fc3c4ada640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:13.748 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.745+0000 7fc3c4ada640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:13.748 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.745+0000 7fc3c4ada640 10 monclient: dump: 2024-03-25T18:13:13.748 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:13.748 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:13.748 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:13.748 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:13.748 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:13.748 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:13.748 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:13.748 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:13.748 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:13.748 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:13.749 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.745+0000 7fc3c7d40640 5 monclient: authenticate success, global_id 4639 2024-03-25T18:13:13.749 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.745+0000 7fc3c7d40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:13.766 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.764+0000 7fc3c4ada640 1 -- 172.21.15.66:0/3875313361 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3bc0017f0 con 0x7fc3c00b4d60 2024-03-25T18:13:13.767 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.764+0000 7fc3c4ada640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:13.767 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.764+0000 7fc3c4ada640 1 -- 172.21.15.66:0/3875313361 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fc3bc001c00 con 0x7fc3c00b4d60 2024-03-25T18:13:13.767 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.764+0000 7fc3c7d40640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:13.767 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.764+0000 7fc3c7d40640 4 set_mon_vals no callback set 2024-03-25T18:13:13.767 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.764+0000 7fc3c4ada640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:13.767 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.764+0000 7fc3c4ada640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:13.767 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.764+0000 7fc3c4ada640 10 monclient: dump: 2024-03-25T18:13:13.767 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:13.767 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:13.767 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:13.768 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:13.768 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:13.768 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:13.768 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:13.768 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:13.768 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:13.768 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:13.768 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.765+0000 7fc3c7d40640 10 monclient: shutdown 2024-03-25T18:13:13.768 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.765+0000 7fc3c7d40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:13.768 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.765+0000 7fc3c7d40640 1 -- 172.21.15.66:0/3875313361 >> v1:172.21.15.193:6789/0 conn(0x7fc3c00b4d60 legacy=0x7fc3c013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:13.768 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.765+0000 7fc3c7d40640 1 -- 172.21.15.66:0/3875313361 shutdown_connections 2024-03-25T18:13:13.768 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.765+0000 7fc3c7d40640 1 -- 172.21.15.66:0/3875313361 >> 172.21.15.66:0/3875313361 conn(0x7fc3c00617a0 msgr2=0x7fc3c0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:13.768 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.765+0000 7fc3c7d40640 1 -- 172.21.15.66:0/3875313361 shutdown_connections 2024-03-25T18:13:13.769 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.765+0000 7fc3c7d40640 1 -- 172.21.15.66:0/3875313361 wait complete. 2024-03-25T18:13:13.770 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.766+0000 7fc3c7d40640 10 monclient: build_initial_monmap 2024-03-25T18:13:13.770 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.766+0000 7fc3c7d40640 10 monclient: monmap: 2024-03-25T18:13:13.770 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:13:13.770 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:13.770 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:13:13.770 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:13:13.770 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:13.770 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:13:13.771 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:13:13.771 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:13:13.771 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:13:13.771 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:13.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.766+0000 7fc3c7d40640 1 librados: starting msgr at 2024-03-25T18:13:13.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.766+0000 7fc3c7d40640 1 librados: starting objecter 2024-03-25T18:13:13.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.766+0000 7fc3c7d40640 1 Processor -- start 2024-03-25T18:13:13.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.766+0000 7fc3c7d40640 1 -- start start 2024-03-25T18:13:13.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.766+0000 7fc3c7d40640 1 librados: setting wanted keys 2024-03-25T18:13:13.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.766+0000 7fc3c7d40640 1 librados: calling monclient init 2024-03-25T18:13:13.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.766+0000 7fc3c7d40640 10 monclient: init 2024-03-25T18:13:13.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.767+0000 7fc3c7d40640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:13.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.767+0000 7fc3c7d40640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:13.772 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.767+0000 7fc3c7d40640 10 monclient(hunting): picked mon.noname-c con 0x7fc3c00b1730 addr v1:172.21.15.66:6790/0 2024-03-25T18:13:13.772 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.767+0000 7fc3c7d40640 10 monclient(hunting): picked mon.noname-b con 0x7fc3c00b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T18:13:13.772 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.767+0000 7fc3c7d40640 10 monclient(hunting): picked mon.noname-a con 0x7fc3c014be90 addr v1:172.21.15.66:6789/0 2024-03-25T18:13:13.772 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.767+0000 7fc3c7d40640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c013b6a0 con 0x7fc3c014be90 2024-03-25T18:13:13.772 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.767+0000 7fc3c7d40640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c0137070 con 0x7fc3c00b4d60 2024-03-25T18:13:13.772 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.767+0000 7fc3c7d40640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c0134a00 con 0x7fc3c00b1730 2024-03-25T18:13:13.772 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.767+0000 7fc3c7d40640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:13.772 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.767+0000 7fc3c7d40640 10 monclient(hunting): authenticate will time out at 7542.827409s 2024-03-25T18:13:13.773 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.770+0000 7fc3c62dd640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7fc3c014be90 0x7fc3c01342f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:60362/0 (socket says 172.21.15.66:60362) 2024-03-25T18:13:13.773 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.770+0000 7fc3c62dd640 1 -- 172.21.15.66:0/1980329212 learned_addr learned my addr 172.21.15.66:0/1980329212 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:13:13.773 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.771+0000 7fc3b67fc640 10 client.?.objecter ms_handle_connect 0x7fc3c014be90 2024-03-25T18:13:13.773 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.771+0000 7fc3b67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:13.780 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.778+0000 7fc3b67fc640 10 client.?.objecter ms_handle_connect 0x7fc3c00b4d60 2024-03-25T18:13:13.780 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.778+0000 7fc3b67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:13.785 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.784+0000 7fc3b67fc640 10 client.?.objecter ms_handle_connect 0x7fc3c00b1730 2024-03-25T18:13:13.786 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.784+0000 7fc3b67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:13.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.789+0000 7fc3b67fc640 1 -- 172.21.15.66:0/1980329212 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 786498507 0 0) 0x7fc3c013b6a0 con 0x7fc3c014be90 2024-03-25T18:13:13.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.789+0000 7fc3b67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:13.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.789+0000 7fc3b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:13.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.789+0000 7fc3b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:13.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.789+0000 7fc3b67fc640 10 monclient(hunting): my global_id is 4647 2024-03-25T18:13:13.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.789+0000 7fc3b67fc640 1 -- 172.21.15.66:0/1980329212 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3a4004eb0 con 0x7fc3c014be90 2024-03-25T18:13:13.797 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.796+0000 7fc3b67fc640 1 -- 172.21.15.66:0/1980329212 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1377080008 0 0) 0x7fc3c0137070 con 0x7fc3c00b4d60 2024-03-25T18:13:13.797 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.796+0000 7fc3b67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:13.797 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.796+0000 7fc3b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:13.798 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.796+0000 7fc3b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:13.798 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.796+0000 7fc3b67fc640 10 monclient(hunting): my global_id is 4642 2024-03-25T18:13:13.798 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.796+0000 7fc3b67fc640 1 -- 172.21.15.66:0/1980329212 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c013b6a0 con 0x7fc3c00b4d60 2024-03-25T18:13:13.805 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.804+0000 7fc3b67fc640 1 -- 172.21.15.66:0/1980329212 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457528057 0 0) 0x7fc3c0134a00 con 0x7fc3c00b1730 2024-03-25T18:13:13.805 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.804+0000 7fc3b67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:13.806 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.804+0000 7fc3b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:13.806 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.804+0000 7fc3b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:13.806 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.804+0000 7fc3b67fc640 10 monclient(hunting): my global_id is 4643 2024-03-25T18:13:13.806 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.804+0000 7fc3b67fc640 1 -- 172.21.15.66:0/1980329212 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c0137070 con 0x7fc3c00b1730 2024-03-25T18:13:13.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.813+0000 7fc3b67fc640 1 -- 172.21.15.66:0/1980329212 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 868436988 0 0) 0x7fc3c013b6a0 con 0x7fc3c00b4d60 2024-03-25T18:13:13.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.813+0000 7fc3b67fc640 1 -- 172.21.15.66:0/1980329212 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3c0134a00 con 0x7fc3c00b4d60 2024-03-25T18:13:13.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.813+0000 7fc3b67fc640 1 -- 172.21.15.66:0/1980329212 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fc3b00018b0 con 0x7fc3c00b4d60 2024-03-25T18:13:13.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.813+0000 7fc3b67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:13.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.813+0000 7fc3b67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:13.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.813+0000 7fc3b67fc640 10 monclient(hunting): dump: 2024-03-25T18:13:13.815 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:13.815 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:13.815 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:13.815 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:13.815 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:13.815 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:13.815 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:13.816 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:13.816 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:13.816 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:13.816 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.813+0000 7fc3b67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:13.817 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.815+0000 7fc3b67fc640 1 -- 172.21.15.66:0/1980329212 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2705865591 0 0) 0x7fc3a4004eb0 con 0x7fc3c014be90 2024-03-25T18:13:13.817 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.815+0000 7fc3b67fc640 1 -- 172.21.15.66:0/1980329212 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3c013b6a0 con 0x7fc3c014be90 2024-03-25T18:13:13.817 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.815+0000 7fc3b67fc640 1 -- 172.21.15.66:0/1980329212 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fc3b8001c70 con 0x7fc3c014be90 2024-03-25T18:13:13.817 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.815+0000 7fc3b67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:13.817 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.815+0000 7fc3b67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:13.817 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.815+0000 7fc3b67fc640 10 monclient(hunting): dump: 2024-03-25T18:13:13.817 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:13.817 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:13.817 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:13.817 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:13.817 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:13.817 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:13.817 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:13.818 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:13.818 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:13.818 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:13.829 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.827+0000 7fc3b67fc640 1 -- 172.21.15.66:0/1980329212 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1698764904 0 0) 0x7fc3c0134a00 con 0x7fc3c00b4d60 2024-03-25T18:13:13.829 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.827+0000 7fc3b67fc640 1 -- 172.21.15.66:0/1980329212 >> v1:172.21.15.66:6790/0 conn(0x7fc3c00b1730 legacy=0x7fc3c0137cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:13.829 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.827+0000 7fc3b67fc640 1 -- 172.21.15.66:0/1980329212 >> v1:172.21.15.66:6789/0 conn(0x7fc3c014be90 legacy=0x7fc3c01342f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:13.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.827+0000 7fc3b67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:13:13.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.827+0000 7fc3b67fc640 1 monclient: found mon.b 2024-03-25T18:13:13.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.827+0000 7fc3b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:13.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.827+0000 7fc3b67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:13:13.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.827+0000 7fc3b67fc640 1 -- 172.21.15.66:0/1980329212 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c01372e0 con 0x7fc3c00b4d60 2024-03-25T18:13:13.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.827+0000 7fc3b67fc640 10 monclient: _finish_auth 0 2024-03-25T18:13:13.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.827+0000 7fc3b67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:13:13.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.827+0000 7fc3b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:13.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.827+0000 7fc3c7d40640 5 monclient: authenticate success, global_id 4642 2024-03-25T18:13:13.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.827+0000 7fc3c7d40640 10 monclient: _renew_subs 2024-03-25T18:13:13.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.827+0000 7fc3c7d40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:13:13.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.827+0000 7fc3c7d40640 1 -- 172.21.15.66:0/1980329212 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3c01375c0 con 0x7fc3c00b4d60 2024-03-25T18:13:13.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.828+0000 7fc3c7d40640 10 client.4642.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:13.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.828+0000 7fc3c7d40640 10 monclient: _renew_subs 2024-03-25T18:13:13.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.828+0000 7fc3c7d40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:13:13.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.828+0000 7fc3c7d40640 1 -- 172.21.15.66:0/1980329212 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3c01383d0 con 0x7fc3c00b4d60 2024-03-25T18:13:13.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.828+0000 7fc3c7d40640 1 librados: init done 2024-03-25T18:13:13.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.829+0000 7fc3c7d40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:13.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.829+0000 7fc3c7d40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:13.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.829+0000 7fc3c7d40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:13:13.832 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:13.829+0000 7fc3c7d40640 1 -- 172.21.15.66:0/1980329212 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3c01379b0 con 0x7fc3c00b4d60 2024-03-25T18:13:14.113 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.112+0000 7fc3b67fc640 1 -- 172.21.15.66:0/1980329212 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3b00035f0 con 0x7fc3c00b4d60 2024-03-25T18:13:14.114 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.112+0000 7fc3b67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:14.114 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.112+0000 7fc3b67fc640 1 -- 172.21.15.66:0/1980329212 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fc3b00039c0 con 0x7fc3c00b4d60 2024-03-25T18:13:14.114 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.112+0000 7fc3b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:14.115 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.112+0000 7fc3b67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:14.115 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.112+0000 7fc3b67fc640 10 monclient: dump: 2024-03-25T18:13:14.115 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:14.115 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:14.115 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:14.115 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:14.115 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:14.115 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:14.115 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:14.115 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:14.115 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:14.115 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:14.115 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.112+0000 7fc3b7fff640 4 set_mon_vals no callback set 2024-03-25T18:13:14.116 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.113+0000 7fc3b67fc640 1 -- 172.21.15.66:0/1980329212 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7fc3b0003be0 con 0x7fc3c00b4d60 2024-03-25T18:13:14.116 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.113+0000 7fc3c5adc640 10 monclient: get_auth_request con 0x7fc3a406df70 auth_method 0 2024-03-25T18:13:14.116 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.114+0000 7fc3b67fc640 10 client.4642.objecter ms_handle_connect 0x7fc3a406df70 2024-03-25T18:13:14.120 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.118+0000 7fc3b67fc640 1 -- 172.21.15.66:0/1980329212 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7fc3b00535a0 con 0x7fc3c00b4d60 2024-03-25T18:13:14.120 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.118+0000 7fc3b67fc640 10 client.4642.objecter ms_dispatch 0x7fc3c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:14.120 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.118+0000 7fc3b67fc640 3 client.4642.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:14.120 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.118+0000 7fc3b67fc640 3 client.4642.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:14.120 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.118+0000 7fc3b67fc640 20 client.4642.objecter dump_active .. 0 homeless 2024-03-25T18:13:14.120 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.118+0000 7fc3b67fc640 1 -- 172.21.15.66:0/1980329212 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc3b004f0b0 con 0x7fc3c00b4d60 2024-03-25T18:13:14.120 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.118+0000 7fc3b67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:14.120 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.118+0000 7fc3b67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:14.381 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.380+0000 7fc3c7d40640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:14.381 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.380+0000 7fc3c7d40640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:14.382 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.380+0000 7fc3c7d40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:13:14.382 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.380+0000 7fc3c7d40640 1 -- 172.21.15.66:0/1980329212 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3c013b850 con 0x7fc3c00b4d60 2024-03-25T18:13:14.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.405+0000 7fc3b67fc640 1 -- 172.21.15.66:0/1980329212 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7fc3c013b850 con 0x7fc3c00b4d60 2024-03-25T18:13:14.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.406+0000 7fc3b67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:14.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.406+0000 7fc3b67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:14.407 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:13:14.408 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:14.410 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.408+0000 7fc3c7d40640 10 librados: watch_flush enter 2024-03-25T18:13:14.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.408+0000 7fc3c7d40640 10 librados: watch_flush exit 2024-03-25T18:13:14.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.408+0000 7fc3c7d40640 20 client.4642.objecter shutdown clearing up homeless session... 2024-03-25T18:13:14.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.408+0000 7fc3c7d40640 10 client.4642.objecter successfully canceled tick 2024-03-25T18:13:14.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.408+0000 7fc3c7d40640 1 -- 172.21.15.66:0/1980329212 >> v1:172.21.15.193:6800/2275676803 conn(0x7fc3a406df70 legacy=0x7fc3a4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:14.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.408+0000 7fc3c7d40640 10 monclient: shutdown 2024-03-25T18:13:14.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.408+0000 7fc3c7d40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:14.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.408+0000 7fc3c7d40640 1 -- 172.21.15.66:0/1980329212 >> v1:172.21.15.193:6789/0 conn(0x7fc3c00b4d60 legacy=0x7fc3c0133be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:14.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.408+0000 7fc3c7d40640 1 -- 172.21.15.66:0/1980329212 shutdown_connections 2024-03-25T18:13:14.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.408+0000 7fc3c7d40640 1 -- 172.21.15.66:0/1980329212 >> 172.21.15.66:0/1980329212 conn(0x7fc3c00617a0 msgr2=0x7fc3c0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:14.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.408+0000 7fc3c7d40640 1 -- 172.21.15.66:0/1980329212 shutdown_connections 2024-03-25T18:13:14.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.408+0000 7fc3c7d40640 1 -- 172.21.15.66:0/1980329212 wait complete. 2024-03-25T18:13:14.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:14.409+0000 7fc3c7d40640 1 librados: shutdown 2024-03-25T18:13:14.423 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:21.229 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:21.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.350+0000 7f2206b0e640 10 monclient: get_monmap_and_config 2024-03-25T18:13:21.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.350+0000 7f2206b0e640 10 monclient: build_initial_monmap 2024-03-25T18:13:21.352 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.350+0000 7f2206b0e640 10 monclient: monmap: 2024-03-25T18:13:21.352 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:13:21.352 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:21.352 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:13:21.352301+0000 2024-03-25T18:13:21.352 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:13:21.352301+0000 2024-03-25T18:13:21.352 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:21.352 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:21.353 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:13:21.353 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:13:21.353 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:13:21.353 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:21.353 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.351+0000 7f2206b0e640 1 Processor -- start 2024-03-25T18:13:21.353 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.351+0000 7f2206b0e640 1 -- start start 2024-03-25T18:13:21.353 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.351+0000 7f2206b0e640 10 monclient: init 2024-03-25T18:13:21.354 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.352+0000 7f2206b0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:21.354 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.352+0000 7f2206b0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:13:21.355 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.352+0000 7f2206b0e640 10 monclient(hunting): picked mon.noname-c con 0x7f22000a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:13:21.355 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.352+0000 7f2206b0e640 10 monclient(hunting): picked mon.noname-b con 0x7f220013f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:13:21.355 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.352+0000 7f2206b0e640 10 monclient(hunting): picked mon.noname-a con 0x7f22001430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:13:21.355 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.352+0000 7f2206b0e640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2200142520 con 0x7f22001430c0 2024-03-25T18:13:21.355 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.352+0000 7f2206b0e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22000b28a0 con 0x7f22000a98f0 2024-03-25T18:13:21.355 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.352+0000 7f2206b0e640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22000b2a30 con 0x7f220013f710 2024-03-25T18:13:21.355 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.352+0000 7f2206b0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:21.355 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.352+0000 7f2206b0e640 10 monclient(hunting): authenticate will time out at 7550.412969s 2024-03-25T18:13:21.365 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.363+0000 7f22048aa640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f22000a98f0 0x7f22000a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:33838/0 (socket says 172.21.15.66:33838) 2024-03-25T18:13:21.365 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.364+0000 7f22048aa640 1 -- 172.21.15.66:0/2394126989 learned_addr learned my addr 172.21.15.66:0/2394126989 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:13:21.376 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.374+0000 7f21ff7fe640 1 -- 172.21.15.66:0/2394126989 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2057553766 0 0) 0x7f22000b28a0 con 0x7f22000a98f0 2024-03-25T18:13:21.376 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.374+0000 7f21ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:21.376 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.374+0000 7f21ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:21.376 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.374+0000 7f21ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:21.376 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.374+0000 7f21ff7fe640 10 monclient(hunting): my global_id is 4645 2024-03-25T18:13:21.376 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.374+0000 7f21ff7fe640 1 -- 172.21.15.66:0/2394126989 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21f0003520 con 0x7f22000a98f0 2024-03-25T18:13:21.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.377+0000 7f21ff7fe640 1 -- 172.21.15.66:0/2394126989 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2084185499 0 0) 0x7f22000b2a30 con 0x7f220013f710 2024-03-25T18:13:21.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.377+0000 7f21ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:21.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.377+0000 7f21ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:21.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.377+0000 7f21ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:21.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.377+0000 7f21ff7fe640 10 monclient(hunting): my global_id is 4646 2024-03-25T18:13:21.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.377+0000 7f21ff7fe640 1 -- 172.21.15.66:0/2394126989 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22000b28a0 con 0x7f220013f710 2024-03-25T18:13:21.385 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.383+0000 7f21ff7fe640 1 -- 172.21.15.66:0/2394126989 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 785074976 0 0) 0x7f21f0003520 con 0x7f22000a98f0 2024-03-25T18:13:21.385 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.383+0000 7f21ff7fe640 1 -- 172.21.15.66:0/2394126989 >> v1:172.21.15.66:6790/0 conn(0x7f220013f710 legacy=0x7f22001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:21.385 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.383+0000 7f21ff7fe640 1 -- 172.21.15.66:0/2394126989 >> v1:172.21.15.66:6789/0 conn(0x7f22001430c0 legacy=0x7f220014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:21.385 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.383+0000 7f21ff7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:13:21.385 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.383+0000 7f21ff7fe640 1 monclient: found mon.noname-c 2024-03-25T18:13:21.385 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.383+0000 7f21ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:21.385 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.383+0000 7f21ff7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:13:21.386 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.383+0000 7f21ff7fe640 1 -- 172.21.15.66:0/2394126989 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22000b44c0 con 0x7f22000a98f0 2024-03-25T18:13:21.386 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.383+0000 7f21ff7fe640 10 monclient: _finish_auth 0 2024-03-25T18:13:21.386 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.383+0000 7f21ff7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:13:21.386 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.383+0000 7f21ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:21.386 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.383+0000 7f21ff7fe640 1 -- 172.21.15.66:0/2394126989 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f21f4001690 con 0x7f22000a98f0 2024-03-25T18:13:21.386 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.383+0000 7f2206b0e640 5 monclient: authenticate success, global_id 4645 2024-03-25T18:13:21.386 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.383+0000 7f2206b0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:21.386 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.383+0000 7f21ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:21.386 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.383+0000 7f21ff7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:21.386 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.383+0000 7f21ff7fe640 10 monclient: dump: 2024-03-25T18:13:21.386 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:21.386 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:21.386 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:21.386 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:21.387 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:21.387 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:21.387 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:21.387 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:21.387 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:21.387 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:21.387 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.383+0000 7f2206b0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:21.419 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.418+0000 7f21ff7fe640 1 -- 172.21.15.66:0/2394126989 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21f40017f0 con 0x7f22000a98f0 2024-03-25T18:13:21.419 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.418+0000 7f21ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:21.420 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.418+0000 7f21ff7fe640 1 -- 172.21.15.66:0/2394126989 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f21f4001c00 con 0x7f22000a98f0 2024-03-25T18:13:21.420 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.418+0000 7f2206b0e640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:21.420 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.418+0000 7f2206b0e640 4 set_mon_vals no callback set 2024-03-25T18:13:21.420 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.418+0000 7f21ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:21.420 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.418+0000 7f21ff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:21.420 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.418+0000 7f21ff7fe640 10 monclient: dump: 2024-03-25T18:13:21.420 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:21.420 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:21.420 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:21.420 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:21.420 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:21.420 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:21.420 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:21.420 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:21.420 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:21.420 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:21.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.419+0000 7f2206b0e640 10 monclient: shutdown 2024-03-25T18:13:21.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.419+0000 7f2206b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:21.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.419+0000 7f2206b0e640 1 -- 172.21.15.66:0/2394126989 >> v1:172.21.15.193:6789/0 conn(0x7f22000a98f0 legacy=0x7f22000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:21.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.419+0000 7f2206b0e640 1 -- 172.21.15.66:0/2394126989 shutdown_connections 2024-03-25T18:13:21.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.419+0000 7f2206b0e640 1 -- 172.21.15.66:0/2394126989 >> 172.21.15.66:0/2394126989 conn(0x7f22000b7ea0 msgr2=0x7f22000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:21.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.419+0000 7f2206b0e640 1 -- 172.21.15.66:0/2394126989 shutdown_connections 2024-03-25T18:13:21.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.419+0000 7f2206b0e640 1 -- 172.21.15.66:0/2394126989 wait complete. 2024-03-25T18:13:21.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.420+0000 7f2206b0e640 10 monclient: build_initial_monmap 2024-03-25T18:13:21.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.420+0000 7f2206b0e640 10 monclient: monmap: 2024-03-25T18:13:21.422 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:13:21.422 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:21.422 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:13:21.422 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:13:21.423 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:21.423 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:13:21.423 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:13:21.423 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:13:21.423 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:13:21.423 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:21.423 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.420+0000 7f2206b0e640 1 librados: starting msgr at 2024-03-25T18:13:21.423 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.420+0000 7f2206b0e640 1 librados: starting objecter 2024-03-25T18:13:21.423 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.420+0000 7f2206b0e640 1 Processor -- start 2024-03-25T18:13:21.423 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.420+0000 7f2206b0e640 1 -- start start 2024-03-25T18:13:21.423 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.420+0000 7f2206b0e640 1 librados: setting wanted keys 2024-03-25T18:13:21.423 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.420+0000 7f2206b0e640 1 librados: calling monclient init 2024-03-25T18:13:21.423 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.420+0000 7f2206b0e640 10 monclient: init 2024-03-25T18:13:21.424 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.421+0000 7f2206b0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:21.424 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.421+0000 7f2206b0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:13:21.424 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.421+0000 7f2206b0e640 10 monclient(hunting): picked mon.noname-b con 0x7f22000a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:13:21.424 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.421+0000 7f2206b0e640 10 monclient(hunting): picked mon.noname-c con 0x7f220013f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:13:21.425 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.421+0000 7f2206b0e640 10 monclient(hunting): picked mon.noname-a con 0x7f22001430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:13:21.425 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.421+0000 7f2206b0e640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2200135730 con 0x7f22001430c0 2024-03-25T18:13:21.425 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.421+0000 7f2206b0e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22001358c0 con 0x7f22000a98f0 2024-03-25T18:13:21.425 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.421+0000 7f2206b0e640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2200135a50 con 0x7f220013f710 2024-03-25T18:13:21.425 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.421+0000 7f2206b0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:21.425 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.421+0000 7f2206b0e640 10 monclient(hunting): authenticate will time out at 7550.481652s 2024-03-25T18:13:21.434 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.433+0000 7f21fffff640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f220013f710 0x7f22001d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:52950/0 (socket says 172.21.15.66:52950) 2024-03-25T18:13:21.434 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.433+0000 7f21fffff640 1 -- 172.21.15.66:0/1357969463 learned_addr learned my addr 172.21.15.66:0/1357969463 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:13:21.439 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.438+0000 7f21fd7fa640 10 client.?.objecter ms_handle_connect 0x7f22000a98f0 2024-03-25T18:13:21.439 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.438+0000 7f21fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:21.440 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.438+0000 7f21fd7fa640 10 client.?.objecter ms_handle_connect 0x7f220013f710 2024-03-25T18:13:21.440 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.438+0000 7f21fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:21.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.446+0000 7f21fd7fa640 1 -- 172.21.15.66:0/1357969463 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 945062422 0 0) 0x7f2200135a50 con 0x7f220013f710 2024-03-25T18:13:21.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.446+0000 7f21fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:13:21.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.446+0000 7f21fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:21.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.446+0000 7f21fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:21.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.446+0000 7f21fd7fa640 10 monclient(hunting): my global_id is 4649 2024-03-25T18:13:21.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.446+0000 7f21fd7fa640 1 -- 172.21.15.66:0/1357969463 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21e4004eb0 con 0x7f220013f710 2024-03-25T18:13:21.458 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.457+0000 7f21fd7fa640 10 client.?.objecter ms_handle_connect 0x7f22001430c0 2024-03-25T18:13:21.458 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.457+0000 7f21fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:21.458 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.457+0000 7f21fd7fa640 1 -- 172.21.15.66:0/1357969463 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919359042 0 0) 0x7f22001358c0 con 0x7f22000a98f0 2024-03-25T18:13:21.458 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.457+0000 7f21fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:13:21.458 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.457+0000 7f21fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:21.459 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.457+0000 7f21fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:21.459 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.457+0000 7f21fd7fa640 10 monclient(hunting): my global_id is 4648 2024-03-25T18:13:21.459 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.457+0000 7f21fd7fa640 1 -- 172.21.15.66:0/1357969463 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2200135a50 con 0x7f22000a98f0 2024-03-25T18:13:21.466 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.463+0000 7f21fd7fa640 1 -- 172.21.15.66:0/1357969463 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4031847397 0 0) 0x7f2200135730 con 0x7f22001430c0 2024-03-25T18:13:21.466 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.463+0000 7f21fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:13:21.466 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.463+0000 7f21fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:21.466 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.463+0000 7f21fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:21.466 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.463+0000 7f21fd7fa640 10 monclient(hunting): my global_id is 4653 2024-03-25T18:13:21.466 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.463+0000 7f21fd7fa640 1 -- 172.21.15.66:0/1357969463 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22001358c0 con 0x7f22001430c0 2024-03-25T18:13:21.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.466+0000 7f21fd7fa640 1 -- 172.21.15.66:0/1357969463 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1425626253 0 0) 0x7f21e4004eb0 con 0x7f220013f710 2024-03-25T18:13:21.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.466+0000 7f21fd7fa640 1 -- 172.21.15.66:0/1357969463 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2200135730 con 0x7f220013f710 2024-03-25T18:13:21.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.466+0000 7f21fd7fa640 1 -- 172.21.15.66:0/1357969463 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f21ec001a10 con 0x7f220013f710 2024-03-25T18:13:21.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.466+0000 7f21fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:21.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.466+0000 7f21fd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:21.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.466+0000 7f21fd7fa640 10 monclient(hunting): dump: 2024-03-25T18:13:21.467 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:21.468 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:21.468 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:21.468 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:21.468 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:21.468 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:21.468 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:21.468 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:21.468 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:21.468 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:21.468 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.466+0000 7f21fd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:21.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.476+0000 7f21fd7fa640 1 -- 172.21.15.66:0/1357969463 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1035138322 0 0) 0x7f22001358c0 con 0x7f22001430c0 2024-03-25T18:13:21.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.476+0000 7f21fd7fa640 1 -- 172.21.15.66:0/1357969463 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21e4004eb0 con 0x7f22001430c0 2024-03-25T18:13:21.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.476+0000 7f21fd7fa640 1 -- 172.21.15.66:0/1357969463 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f21f8001a60 con 0x7f22001430c0 2024-03-25T18:13:21.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.476+0000 7f21fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:21.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.476+0000 7f21fd7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:21.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.476+0000 7f21fd7fa640 10 monclient(hunting): dump: 2024-03-25T18:13:21.478 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:21.478 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:21.478 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:21.478 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:21.478 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:21.479 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:21.479 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:21.479 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:21.479 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:21.479 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:21.480 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.478+0000 7f21fd7fa640 1 -- 172.21.15.66:0/1357969463 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1850433485 0 0) 0x7f2200135a50 con 0x7f22000a98f0 2024-03-25T18:13:21.480 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.478+0000 7f21fd7fa640 1 -- 172.21.15.66:0/1357969463 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22001358c0 con 0x7f22000a98f0 2024-03-25T18:13:21.480 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.478+0000 7f21fd7fa640 1 -- 172.21.15.66:0/1357969463 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f21f4001b40 con 0x7f22000a98f0 2024-03-25T18:13:21.480 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.478+0000 7f21fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:21.480 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.478+0000 7f21fd7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:21.480 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.478+0000 7f21fd7fa640 10 monclient(hunting): dump: 2024-03-25T18:13:21.480 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:21.480 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:21.480 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:21.480 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:21.480 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:21.480 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:21.480 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:21.480 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:21.480 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:21.481 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:21.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.484+0000 7f21fd7fa640 1 -- 172.21.15.66:0/1357969463 <== mon.0 v1:172.21.15.66:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 229129068 0 0) 0x7f21e4004eb0 con 0x7f22001430c0 2024-03-25T18:13:21.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.484+0000 7f21fd7fa640 1 -- 172.21.15.66:0/1357969463 >> v1:172.21.15.66:6790/0 conn(0x7f220013f710 legacy=0x7f22001d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:21.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.484+0000 7f21fd7fa640 1 -- 172.21.15.66:0/1357969463 >> v1:172.21.15.193:6789/0 conn(0x7f22000a98f0 legacy=0x7f22001dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:21.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.484+0000 7f21fd7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:13:21.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.484+0000 7f21fd7fa640 1 monclient: found mon.a 2024-03-25T18:13:21.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.484+0000 7f21fd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:21.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.484+0000 7f21fd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:13:21.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.484+0000 7f21fd7fa640 1 -- 172.21.15.66:0/1357969463 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22001d5000 con 0x7f22001430c0 2024-03-25T18:13:21.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.484+0000 7f21fd7fa640 10 monclient: _finish_auth 0 2024-03-25T18:13:21.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.484+0000 7f21fd7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:13:21.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.484+0000 7f21fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:21.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.484+0000 7f2206b0e640 5 monclient: authenticate success, global_id 4653 2024-03-25T18:13:21.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.485+0000 7f2206b0e640 10 monclient: _renew_subs 2024-03-25T18:13:21.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.485+0000 7f2206b0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:13:21.488 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.485+0000 7f2206b0e640 1 -- 172.21.15.66:0/1357969463 --> v1:172.21.15.66:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22001d52b0 con 0x7f22001430c0 2024-03-25T18:13:21.488 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.485+0000 7f2206b0e640 10 client.4653.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:21.488 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.485+0000 7f2206b0e640 10 monclient: _renew_subs 2024-03-25T18:13:21.488 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.485+0000 7f2206b0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:13:21.488 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.485+0000 7f2206b0e640 1 -- 172.21.15.66:0/1357969463 --> v1:172.21.15.66:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f220013c900 con 0x7f22001430c0 2024-03-25T18:13:21.488 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.485+0000 7f2206b0e640 1 librados: init done 2024-03-25T18:13:21.489 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.486+0000 7f2206b0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:21.489 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.486+0000 7f2206b0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:21.489 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.486+0000 7f2206b0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:13:21.489 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.486+0000 7f2206b0e640 1 -- 172.21.15.66:0/1357969463 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f220013cbb0 con 0x7f22001430c0 2024-03-25T18:13:21.802 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.800+0000 7f21fd7fa640 1 -- 172.21.15.66:0/1357969463 <== mon.0 v1:172.21.15.66:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21f8001f90 con 0x7f22001430c0 2024-03-25T18:13:21.802 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.800+0000 7f21fd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:21.802 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.800+0000 7f21fd7fa640 1 -- 172.21.15.66:0/1357969463 <== mon.0 v1:172.21.15.66:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f21f8003970 con 0x7f22001430c0 2024-03-25T18:13:21.803 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.800+0000 7f21fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:21.803 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.800+0000 7f21fd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:21.803 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.800+0000 7f21fd7fa640 10 monclient: dump: 2024-03-25T18:13:21.803 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:21.803 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:21.803 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:21.803 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:21.803 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:21.803 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:21.803 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:21.803 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:21.803 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:21.803 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:21.803 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.800+0000 7f21feffd640 4 set_mon_vals no callback set 2024-03-25T18:13:21.804 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.801+0000 7f21fd7fa640 1 -- 172.21.15.66:0/1357969463 <== mon.0 v1:172.21.15.66:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f21f8019940 con 0x7f22001430c0 2024-03-25T18:13:21.804 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.802+0000 7f22048aa640 10 monclient: get_auth_request con 0x7f21e406e110 auth_method 0 2024-03-25T18:13:21.804 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.802+0000 7f21fd7fa640 10 client.4653.objecter ms_handle_connect 0x7f21e406e110 2024-03-25T18:13:21.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.809+0000 7f21fd7fa640 1 -- 172.21.15.66:0/1357969463 <== mon.0 v1:172.21.15.66:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f21f80535e0 con 0x7f22001430c0 2024-03-25T18:13:21.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.809+0000 7f21fd7fa640 10 client.4653.objecter ms_dispatch 0x7f2200000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:21.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.809+0000 7f21fd7fa640 3 client.4653.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:21.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.809+0000 7f21fd7fa640 3 client.4653.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:21.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.809+0000 7f21fd7fa640 20 client.4653.objecter dump_active .. 0 homeless 2024-03-25T18:13:21.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.809+0000 7f21fd7fa640 1 -- 172.21.15.66:0/1357969463 <== mon.0 v1:172.21.15.66:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f21f804f0f0 con 0x7f22001430c0 2024-03-25T18:13:21.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.809+0000 7f21fd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:21.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:21.809+0000 7f21fd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:22.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:22.070+0000 7f2206b0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:22.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:22.070+0000 7f2206b0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:22.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:22.070+0000 7f2206b0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:13:22.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:22.070+0000 7f2206b0e640 1 -- 172.21.15.66:0/1357969463 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22001371c0 con 0x7f22001430c0 2024-03-25T18:13:22.107 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:22.106+0000 7f21fd7fa640 1 -- 172.21.15.66:0/1357969463 <== mon.0 v1:172.21.15.66:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f21f8051020 con 0x7f22001430c0 2024-03-25T18:13:22.108 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:22.106+0000 7f21fd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:22.108 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:22.106+0000 7f21fd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:22.108 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:13:22.108 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:22.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:22.108+0000 7f2206b0e640 10 librados: watch_flush enter 2024-03-25T18:13:22.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:22.108+0000 7f2206b0e640 10 librados: watch_flush exit 2024-03-25T18:13:22.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:22.108+0000 7f2206b0e640 20 client.4653.objecter shutdown clearing up homeless session... 2024-03-25T18:13:22.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:22.108+0000 7f2206b0e640 10 client.4653.objecter successfully canceled tick 2024-03-25T18:13:22.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:22.108+0000 7f2206b0e640 1 -- 172.21.15.66:0/1357969463 >> v1:172.21.15.193:6800/2275676803 conn(0x7f21e406e110 legacy=0x7f21e40705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:22.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:22.108+0000 7f2206b0e640 10 monclient: shutdown 2024-03-25T18:13:22.111 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:22.108+0000 7f2206b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:22.111 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:22.108+0000 7f2206b0e640 1 -- 172.21.15.66:0/1357969463 >> v1:172.21.15.66:6789/0 conn(0x7f22001430c0 legacy=0x7f22001d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:22.111 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:22.108+0000 7f2206b0e640 1 -- 172.21.15.66:0/1357969463 shutdown_connections 2024-03-25T18:13:22.111 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:22.108+0000 7f2206b0e640 1 -- 172.21.15.66:0/1357969463 >> 172.21.15.66:0/1357969463 conn(0x7f22000b7ea0 msgr2=0x7f220014ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:22.111 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:22.108+0000 7f2206b0e640 1 -- 172.21.15.66:0/1357969463 shutdown_connections 2024-03-25T18:13:22.111 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:22.108+0000 7f2206b0e640 1 -- 172.21.15.66:0/1357969463 wait complete. 2024-03-25T18:13:22.111 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:22.108+0000 7f2206b0e640 1 librados: shutdown 2024-03-25T18:13:22.121 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:28.928 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:29.048 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.046+0000 7f3c64655640 10 monclient: get_monmap_and_config 2024-03-25T18:13:29.048 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.046+0000 7f3c64655640 10 monclient: build_initial_monmap 2024-03-25T18:13:29.048 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.046+0000 7f3c64655640 10 monclient: monmap: 2024-03-25T18:13:29.049 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:13:29.049 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:29.049 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:13:29.048458+0000 2024-03-25T18:13:29.049 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:13:29.048458+0000 2024-03-25T18:13:29.049 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:29.049 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:29.049 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:13:29.049 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:13:29.049 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:13:29.049 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:29.050 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.047+0000 7f3c64655640 1 Processor -- start 2024-03-25T18:13:29.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.047+0000 7f3c64655640 1 -- start start 2024-03-25T18:13:29.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.047+0000 7f3c64655640 10 monclient: init 2024-03-25T18:13:29.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.048+0000 7f3c64655640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:29.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.049+0000 7f3c64655640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:29.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.049+0000 7f3c64655640 10 monclient(hunting): picked mon.noname-b con 0x7f3c5c0b4d60 addr v1:172.21.15.66:6790/0 2024-03-25T18:13:29.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.049+0000 7f3c64655640 10 monclient(hunting): picked mon.noname-c con 0x7f3c5c0b1730 addr v1:172.21.15.193:6789/0 2024-03-25T18:13:29.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.049+0000 7f3c64655640 10 monclient(hunting): picked mon.noname-a con 0x7f3c5c14be90 addr v1:172.21.15.66:6789/0 2024-03-25T18:13:29.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.049+0000 7f3c64655640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5c0af940 con 0x7f3c5c14be90 2024-03-25T18:13:29.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.049+0000 7f3c64655640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5c0b0a70 con 0x7f3c5c0b1730 2024-03-25T18:13:29.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.049+0000 7f3c64655640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5c0b0d50 con 0x7f3c5c0b4d60 2024-03-25T18:13:29.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.049+0000 7f3c64655640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:29.051 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.049+0000 7f3c64655640 10 monclient(hunting): authenticate will time out at 7558.109432s 2024-03-25T18:13:29.060 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.059+0000 7f3c61bf0640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f3c5c0b1730 0x7f3c5c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:46370/0 (socket says 172.21.15.66:46370) 2024-03-25T18:13:29.060 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.059+0000 7f3c61bf0640 1 -- 172.21.15.66:0/3353535325 learned_addr learned my addr 172.21.15.66:0/3353535325 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:13:29.068 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.067+0000 7f3c613ef640 1 -- 172.21.15.66:0/3353535325 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 225656363 0 0) 0x7f3c5c0b0a70 con 0x7f3c5c0b1730 2024-03-25T18:13:29.068 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.067+0000 7f3c613ef640 10 monclient(hunting): _negotiate 2024-03-25T18:13:29.068 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.067+0000 7f3c613ef640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:29.068 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.067+0000 7f3c613ef640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:29.069 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.067+0000 7f3c613ef640 10 monclient(hunting): my global_id is 4651 2024-03-25T18:13:29.069 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.067+0000 7f3c613ef640 1 -- 172.21.15.66:0/3353535325 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c48003520 con 0x7f3c5c0b1730 2024-03-25T18:13:29.082 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.080+0000 7f3c613ef640 1 -- 172.21.15.66:0/3353535325 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1000852221 0 0) 0x7f3c48003520 con 0x7f3c5c0b1730 2024-03-25T18:13:29.082 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.080+0000 7f3c613ef640 1 -- 172.21.15.66:0/3353535325 >> v1:172.21.15.66:6790/0 conn(0x7f3c5c0b4d60 legacy=0x7f3c5c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:29.082 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.080+0000 7f3c613ef640 1 -- 172.21.15.66:0/3353535325 >> v1:172.21.15.66:6789/0 conn(0x7f3c5c14be90 legacy=0x7f3c5c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:29.082 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.080+0000 7f3c613ef640 10 monclient: _finish_hunting 0 2024-03-25T18:13:29.082 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.080+0000 7f3c613ef640 1 monclient: found mon.noname-c 2024-03-25T18:13:29.082 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.080+0000 7f3c613ef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:29.082 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.080+0000 7f3c613ef640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:13:29.083 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.080+0000 7f3c613ef640 1 -- 172.21.15.66:0/3353535325 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5c0b1030 con 0x7f3c5c0b1730 2024-03-25T18:13:29.083 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.080+0000 7f3c613ef640 10 monclient: _finish_auth 0 2024-03-25T18:13:29.083 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.080+0000 7f3c613ef640 10 monclient: _check_auth_tickets 2024-03-25T18:13:29.083 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.080+0000 7f3c613ef640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:29.083 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.080+0000 7f3c613ef640 1 -- 172.21.15.66:0/3353535325 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f3c4c001690 con 0x7f3c5c0b1730 2024-03-25T18:13:29.083 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.080+0000 7f3c613ef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:29.083 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.080+0000 7f3c613ef640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:29.083 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.080+0000 7f3c613ef640 10 monclient: dump: 2024-03-25T18:13:29.083 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:29.083 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:29.083 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:29.083 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:29.083 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:29.083 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:29.083 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:29.083 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:29.084 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:29.084 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:29.084 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.080+0000 7f3c64655640 5 monclient: authenticate success, global_id 4651 2024-03-25T18:13:29.084 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.080+0000 7f3c64655640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:29.087 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.085+0000 7f3c613ef640 1 -- 172.21.15.66:0/3353535325 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c4c0017f0 con 0x7f3c5c0b1730 2024-03-25T18:13:29.087 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.085+0000 7f3c613ef640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:29.087 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.085+0000 7f3c613ef640 1 -- 172.21.15.66:0/3353535325 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f3c4c001c00 con 0x7f3c5c0b1730 2024-03-25T18:13:29.087 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.085+0000 7f3c64655640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:29.087 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.085+0000 7f3c64655640 4 set_mon_vals no callback set 2024-03-25T18:13:29.087 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.085+0000 7f3c613ef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:29.087 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.085+0000 7f3c613ef640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:29.087 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.085+0000 7f3c613ef640 10 monclient: dump: 2024-03-25T18:13:29.087 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:29.087 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:29.087 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:29.087 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:29.087 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:29.087 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:29.087 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:29.088 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:29.088 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:29.088 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:29.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.086+0000 7f3c64655640 10 monclient: shutdown 2024-03-25T18:13:29.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.086+0000 7f3c64655640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:29.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.086+0000 7f3c64655640 1 -- 172.21.15.66:0/3353535325 >> v1:172.21.15.193:6789/0 conn(0x7f3c5c0b1730 legacy=0x7f3c5c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:29.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.086+0000 7f3c64655640 1 -- 172.21.15.66:0/3353535325 shutdown_connections 2024-03-25T18:13:29.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.086+0000 7f3c64655640 1 -- 172.21.15.66:0/3353535325 >> 172.21.15.66:0/3353535325 conn(0x7f3c5c0617a0 msgr2=0x7f3c5c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:29.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.086+0000 7f3c64655640 1 -- 172.21.15.66:0/3353535325 shutdown_connections 2024-03-25T18:13:29.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.086+0000 7f3c64655640 1 -- 172.21.15.66:0/3353535325 wait complete. 2024-03-25T18:13:29.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.087+0000 7f3c64655640 10 monclient: build_initial_monmap 2024-03-25T18:13:29.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.087+0000 7f3c64655640 10 monclient: monmap: 2024-03-25T18:13:29.089 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:13:29.090 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:29.090 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:13:29.090 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:13:29.090 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:29.090 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:13:29.090 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:13:29.090 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:13:29.090 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:13:29.090 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:29.090 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.087+0000 7f3c64655640 1 librados: starting msgr at 2024-03-25T18:13:29.090 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.087+0000 7f3c64655640 1 librados: starting objecter 2024-03-25T18:13:29.090 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.087+0000 7f3c64655640 1 Processor -- start 2024-03-25T18:13:29.090 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.087+0000 7f3c64655640 1 -- start start 2024-03-25T18:13:29.090 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.087+0000 7f3c64655640 1 librados: setting wanted keys 2024-03-25T18:13:29.090 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.087+0000 7f3c64655640 1 librados: calling monclient init 2024-03-25T18:13:29.091 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.087+0000 7f3c64655640 10 monclient: init 2024-03-25T18:13:29.091 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.088+0000 7f3c64655640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:29.091 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.088+0000 7f3c64655640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:13:29.091 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.088+0000 7f3c64655640 10 monclient(hunting): picked mon.noname-a con 0x7f3c5c0b1730 addr v1:172.21.15.66:6789/0 2024-03-25T18:13:29.091 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.088+0000 7f3c64655640 10 monclient(hunting): picked mon.noname-c con 0x7f3c5c0b4d60 addr v1:172.21.15.66:6790/0 2024-03-25T18:13:29.092 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.088+0000 7f3c64655640 10 monclient(hunting): picked mon.noname-b con 0x7f3c5c14be90 addr v1:172.21.15.193:6789/0 2024-03-25T18:13:29.092 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.088+0000 7f3c64655640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5c1db590 con 0x7f3c5c0b1730 2024-03-25T18:13:29.092 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.088+0000 7f3c64655640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5c141fc0 con 0x7f3c5c14be90 2024-03-25T18:13:29.092 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.088+0000 7f3c64655640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5c140590 con 0x7f3c5c0b4d60 2024-03-25T18:13:29.092 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.088+0000 7f3c64655640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:29.092 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.088+0000 7f3c64655640 10 monclient(hunting): authenticate will time out at 7558.148738s 2024-03-25T18:13:29.102 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.101+0000 7f3c623f1640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f3c5c0b1730 0x7f3c5c1dae80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:60822/0 (socket says 172.21.15.66:60822) 2024-03-25T18:13:29.102 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.101+0000 7f3c623f1640 1 -- 172.21.15.66:0/441586520 learned_addr learned my addr 172.21.15.66:0/441586520 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:13:29.103 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.101+0000 7f3c52ffd640 10 client.?.objecter ms_handle_connect 0x7f3c5c0b1730 2024-03-25T18:13:29.103 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.101+0000 7f3c52ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:29.106 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.104+0000 7f3c52ffd640 10 client.?.objecter ms_handle_connect 0x7f3c5c0b4d60 2024-03-25T18:13:29.106 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.104+0000 7f3c52ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:29.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.109+0000 7f3c52ffd640 1 -- 172.21.15.66:0/441586520 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551603265 0 0) 0x7f3c5c140590 con 0x7f3c5c0b4d60 2024-03-25T18:13:29.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.109+0000 7f3c52ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:29.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.109+0000 7f3c52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:29.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.109+0000 7f3c52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:29.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.109+0000 7f3c52ffd640 10 monclient(hunting): my global_id is 4655 2024-03-25T18:13:29.110 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.109+0000 7f3c52ffd640 1 -- 172.21.15.66:0/441586520 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c40004eb0 con 0x7f3c5c0b4d60 2024-03-25T18:13:29.112 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.111+0000 7f3c52ffd640 10 client.?.objecter ms_handle_connect 0x7f3c5c14be90 2024-03-25T18:13:29.112 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.111+0000 7f3c52ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:29.122 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.120+0000 7f3c52ffd640 1 -- 172.21.15.66:0/441586520 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 126562928 0 0) 0x7f3c5c1db590 con 0x7f3c5c0b1730 2024-03-25T18:13:29.122 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.120+0000 7f3c52ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:29.122 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.120+0000 7f3c52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:29.122 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.120+0000 7f3c52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:29.122 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.120+0000 7f3c52ffd640 10 monclient(hunting): my global_id is 4659 2024-03-25T18:13:29.122 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.120+0000 7f3c52ffd640 1 -- 172.21.15.66:0/441586520 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c5c140590 con 0x7f3c5c0b1730 2024-03-25T18:13:29.126 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.124+0000 7f3c52ffd640 1 -- 172.21.15.66:0/441586520 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 786750616 0 0) 0x7f3c40004eb0 con 0x7f3c5c0b4d60 2024-03-25T18:13:29.126 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.124+0000 7f3c52ffd640 1 -- 172.21.15.66:0/441586520 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c5c1db590 con 0x7f3c5c0b4d60 2024-03-25T18:13:29.126 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.124+0000 7f3c52ffd640 1 -- 172.21.15.66:0/441586520 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f3c4c001b00 con 0x7f3c5c0b4d60 2024-03-25T18:13:29.126 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.124+0000 7f3c52ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:29.126 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.124+0000 7f3c52ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:29.126 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.124+0000 7f3c52ffd640 10 monclient(hunting): dump: 2024-03-25T18:13:29.126 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:29.126 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:29.127 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:29.127 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:29.127 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:29.127 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:29.127 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:29.127 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:29.127 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:29.127 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:29.127 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.124+0000 7f3c52ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:29.141 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.139+0000 7f3c52ffd640 1 -- 172.21.15.66:0/441586520 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 468141620 0 0) 0x7f3c5c141fc0 con 0x7f3c5c14be90 2024-03-25T18:13:29.141 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.139+0000 7f3c52ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:29.141 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.139+0000 7f3c52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:29.141 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.139+0000 7f3c52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:29.141 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.139+0000 7f3c52ffd640 10 monclient(hunting): my global_id is 4654 2024-03-25T18:13:29.141 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.139+0000 7f3c52ffd640 1 -- 172.21.15.66:0/441586520 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c40004eb0 con 0x7f3c5c14be90 2024-03-25T18:13:29.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.142+0000 7f3c52ffd640 1 -- 172.21.15.66:0/441586520 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 747913503 0 0) 0x7f3c5c140590 con 0x7f3c5c0b1730 2024-03-25T18:13:29.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.142+0000 7f3c52ffd640 1 -- 172.21.15.66:0/441586520 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c5c141fc0 con 0x7f3c5c0b1730 2024-03-25T18:13:29.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.142+0000 7f3c52ffd640 1 -- 172.21.15.66:0/441586520 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f3c580019b0 con 0x7f3c5c0b1730 2024-03-25T18:13:29.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.142+0000 7f3c52ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:29.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.142+0000 7f3c52ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:29.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.142+0000 7f3c52ffd640 10 monclient(hunting): dump: 2024-03-25T18:13:29.145 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:29.145 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:29.145 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:29.145 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:29.145 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:29.145 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:29.145 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:29.145 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:29.145 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:29.145 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:29.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.151+0000 7f3c52ffd640 1 -- 172.21.15.66:0/441586520 <== mon.2 v1:172.21.15.66:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 802628420 0 0) 0x7f3c5c1db590 con 0x7f3c5c0b4d60 2024-03-25T18:13:29.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.152+0000 7f3c52ffd640 1 -- 172.21.15.66:0/441586520 >> v1:172.21.15.193:6789/0 conn(0x7f3c5c14be90 legacy=0x7f3c5c1e3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:29.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.152+0000 7f3c52ffd640 1 -- 172.21.15.66:0/441586520 >> v1:172.21.15.66:6789/0 conn(0x7f3c5c0b1730 legacy=0x7f3c5c1dae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:29.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.152+0000 7f3c52ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:13:29.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.152+0000 7f3c52ffd640 1 monclient: found mon.c 2024-03-25T18:13:29.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.152+0000 7f3c52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:29.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.152+0000 7f3c52ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:13:29.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.152+0000 7f3c52ffd640 1 -- 172.21.15.66:0/441586520 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5c1d5440 con 0x7f3c5c0b4d60 2024-03-25T18:13:29.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.152+0000 7f3c52ffd640 10 monclient: _finish_auth 0 2024-03-25T18:13:29.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.152+0000 7f3c52ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:13:29.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.152+0000 7f3c52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:29.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.152+0000 7f3c64655640 5 monclient: authenticate success, global_id 4655 2024-03-25T18:13:29.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.152+0000 7f3c64655640 10 monclient: _renew_subs 2024-03-25T18:13:29.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.152+0000 7f3c64655640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:13:29.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.152+0000 7f3c64655640 1 -- 172.21.15.66:0/441586520 --> v1:172.21.15.66:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c5c135550 con 0x7f3c5c0b4d60 2024-03-25T18:13:29.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.152+0000 7f3c64655640 10 client.4655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:29.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.152+0000 7f3c64655640 10 monclient: _renew_subs 2024-03-25T18:13:29.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.152+0000 7f3c64655640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:13:29.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.152+0000 7f3c64655640 1 -- 172.21.15.66:0/441586520 --> v1:172.21.15.66:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c5c137c40 con 0x7f3c5c0b4d60 2024-03-25T18:13:29.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.152+0000 7f3c64655640 1 librados: init done 2024-03-25T18:13:29.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.153+0000 7f3c64655640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:29.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.153+0000 7f3c64655640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:29.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.153+0000 7f3c64655640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:13:29.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.153+0000 7f3c64655640 1 -- 172.21.15.66:0/441586520 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c5c140720 con 0x7f3c5c0b4d60 2024-03-25T18:13:29.469 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.467+0000 7f3c52ffd640 1 -- 172.21.15.66:0/441586520 <== mon.2 v1:172.21.15.66:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c4c004a90 con 0x7f3c5c0b4d60 2024-03-25T18:13:29.469 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.467+0000 7f3c52ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:29.469 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.467+0000 7f3c52ffd640 1 -- 172.21.15.66:0/441586520 <== mon.2 v1:172.21.15.66:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f3c4c004da0 con 0x7f3c5c0b4d60 2024-03-25T18:13:29.469 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.467+0000 7f3c52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:29.470 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.467+0000 7f3c52ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:29.470 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.467+0000 7f3c52ffd640 10 monclient: dump: 2024-03-25T18:13:29.470 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:29.470 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:29.470 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:29.470 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:29.470 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:29.470 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:29.470 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:29.470 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:29.470 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:29.470 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:29.470 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.467+0000 7f3c60bee640 4 set_mon_vals no callback set 2024-03-25T18:13:29.471 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.468+0000 7f3c52ffd640 1 -- 172.21.15.66:0/441586520 <== mon.2 v1:172.21.15.66:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f3c4c01ad70 con 0x7f3c5c0b4d60 2024-03-25T18:13:29.471 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.469+0000 7f3c623f1640 10 monclient: get_auth_request con 0x7f3c4006dfb0 auth_method 0 2024-03-25T18:13:29.471 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.469+0000 7f3c52ffd640 10 client.4655.objecter ms_handle_connect 0x7f3c4006dfb0 2024-03-25T18:13:29.477 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.475+0000 7f3c52ffd640 1 -- 172.21.15.66:0/441586520 <== mon.2 v1:172.21.15.66:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f3c4c054840 con 0x7f3c5c0b4d60 2024-03-25T18:13:29.477 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.475+0000 7f3c52ffd640 10 client.4655.objecter ms_dispatch 0x7f3c5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:29.477 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.475+0000 7f3c52ffd640 3 client.4655.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:29.477 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.475+0000 7f3c52ffd640 3 client.4655.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:29.477 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.475+0000 7f3c52ffd640 20 client.4655.objecter dump_active .. 0 homeless 2024-03-25T18:13:29.477 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.475+0000 7f3c52ffd640 1 -- 172.21.15.66:0/441586520 <== mon.2 v1:172.21.15.66:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3c4c050350 con 0x7f3c5c0b4d60 2024-03-25T18:13:29.477 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.475+0000 7f3c52ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:29.477 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.475+0000 7f3c52ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:29.713 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.711+0000 7f3c64655640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:29.713 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.711+0000 7f3c64655640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:29.713 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.712+0000 7f3c64655640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:13:29.713 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.712+0000 7f3c64655640 1 -- 172.21.15.66:0/441586520 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c5c0627b0 con 0x7f3c5c0b4d60 2024-03-25T18:13:29.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.793+0000 7f3c52ffd640 1 -- 172.21.15.66:0/441586520 <== mon.2 v1:172.21.15.66:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f3c4c052280 con 0x7f3c5c0b4d60 2024-03-25T18:13:29.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.793+0000 7f3c52ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:29.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.793+0000 7f3c52ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:29.795 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:13:29.795 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:29.797 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.795+0000 7f3c64655640 10 librados: watch_flush enter 2024-03-25T18:13:29.797 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.795+0000 7f3c64655640 10 librados: watch_flush exit 2024-03-25T18:13:29.797 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.795+0000 7f3c64655640 20 client.4655.objecter shutdown clearing up homeless session... 2024-03-25T18:13:29.797 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.795+0000 7f3c64655640 10 client.4655.objecter successfully canceled tick 2024-03-25T18:13:29.797 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.795+0000 7f3c64655640 1 -- 172.21.15.66:0/441586520 >> v1:172.21.15.193:6800/2275676803 conn(0x7f3c4006dfb0 legacy=0x7f3c40070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:29.797 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.795+0000 7f3c64655640 10 monclient: shutdown 2024-03-25T18:13:29.797 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.795+0000 7f3c64655640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:29.797 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.795+0000 7f3c64655640 1 -- 172.21.15.66:0/441586520 >> v1:172.21.15.66:6790/0 conn(0x7f3c5c0b4d60 legacy=0x7f3c5c134e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:29.797 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.795+0000 7f3c64655640 1 -- 172.21.15.66:0/441586520 shutdown_connections 2024-03-25T18:13:29.797 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.795+0000 7f3c64655640 1 -- 172.21.15.66:0/441586520 >> 172.21.15.66:0/441586520 conn(0x7f3c5c0617a0 msgr2=0x7f3c5c13c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:29.798 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.795+0000 7f3c64655640 1 -- 172.21.15.66:0/441586520 shutdown_connections 2024-03-25T18:13:29.798 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.795+0000 7f3c64655640 1 -- 172.21.15.66:0/441586520 wait complete. 2024-03-25T18:13:29.798 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:29.795+0000 7f3c64655640 1 librados: shutdown 2024-03-25T18:13:29.807 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:30.358 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:10:01.339 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:30.366 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:10:01.346 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:30.389 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:10:01.370 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:30.419 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:10:01.378 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:30.423 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:10:01.381 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:30.446 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:10:01.427 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:30.455 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:10:01.413 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:30.458 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:10:01.438 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:30.495 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:10:01.475 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:36.614 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:36.743 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.742+0000 7f3c8606a640 10 monclient: get_monmap_and_config 2024-03-25T18:13:36.743 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.742+0000 7f3c8606a640 10 monclient: build_initial_monmap 2024-03-25T18:13:36.743 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.742+0000 7f3c8606a640 10 monclient: monmap: 2024-03-25T18:13:36.743 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:13:36.743 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:36.743 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:13:36.743551+0000 2024-03-25T18:13:36.744 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:13:36.743551+0000 2024-03-25T18:13:36.744 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:36.744 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:36.744 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:13:36.744 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:13:36.744 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:13:36.744 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:36.745 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.742+0000 7f3c8606a640 1 Processor -- start 2024-03-25T18:13:36.745 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.742+0000 7f3c8606a640 1 -- start start 2024-03-25T18:13:36.745 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.742+0000 7f3c8606a640 10 monclient: init 2024-03-25T18:13:36.745 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.743+0000 7f3c8606a640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:36.745 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.743+0000 7f3c8606a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:13:36.746 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.743+0000 7f3c8606a640 10 monclient(hunting): picked mon.noname-a con 0x7f3c800a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:13:36.746 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.743+0000 7f3c8606a640 10 monclient(hunting): picked mon.noname-b con 0x7f3c8013f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:13:36.746 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.743+0000 7f3c8606a640 10 monclient(hunting): picked mon.noname-c con 0x7f3c801430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:13:36.746 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.743+0000 7f3c8606a640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c80142520 con 0x7f3c800a98f0 2024-03-25T18:13:36.746 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.743+0000 7f3c8606a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c800b28a0 con 0x7f3c801430c0 2024-03-25T18:13:36.746 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.743+0000 7f3c8606a640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c800b2a30 con 0x7f3c8013f710 2024-03-25T18:13:36.746 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.743+0000 7f3c8606a640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:36.746 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.743+0000 7f3c8606a640 10 monclient(hunting): authenticate will time out at 7565.803803s 2024-03-25T18:13:36.756 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.754+0000 7f3c7ffff640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f3c801430c0 0x7f3c8014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:48008/0 (socket says 172.21.15.66:48008) 2024-03-25T18:13:36.756 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.754+0000 7f3c7ffff640 1 -- 172.21.15.66:0/3742354796 learned_addr learned my addr 172.21.15.66:0/3742354796 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:13:36.780 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.778+0000 7f3c7e7fc640 1 -- 172.21.15.66:0/3742354796 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1687229796 0 0) 0x7f3c800b28a0 con 0x7f3c801430c0 2024-03-25T18:13:36.781 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.778+0000 7f3c7e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:36.781 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.778+0000 7f3c7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:36.781 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.778+0000 7f3c7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:36.781 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.778+0000 7f3c7e7fc640 10 monclient(hunting): my global_id is 4657 2024-03-25T18:13:36.781 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.778+0000 7f3c7e7fc640 1 -- 172.21.15.66:0/3742354796 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c68003520 con 0x7f3c801430c0 2024-03-25T18:13:36.781 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.779+0000 7f3c7e7fc640 1 -- 172.21.15.66:0/3742354796 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4048317007 0 0) 0x7f3c800b2a30 con 0x7f3c8013f710 2024-03-25T18:13:36.781 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.779+0000 7f3c7e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:36.781 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.779+0000 7f3c7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:36.781 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.779+0000 7f3c7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:36.781 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.779+0000 7f3c7e7fc640 10 monclient(hunting): my global_id is 4658 2024-03-25T18:13:36.781 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.779+0000 7f3c7e7fc640 1 -- 172.21.15.66:0/3742354796 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c800b28a0 con 0x7f3c8013f710 2024-03-25T18:13:36.782 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.780+0000 7f3c7e7fc640 1 -- 172.21.15.66:0/3742354796 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627509405 0 0) 0x7f3c80142520 con 0x7f3c800a98f0 2024-03-25T18:13:36.782 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.780+0000 7f3c7e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:36.782 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.780+0000 7f3c7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:36.782 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.780+0000 7f3c7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:36.782 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.780+0000 7f3c7e7fc640 10 monclient(hunting): my global_id is 4662 2024-03-25T18:13:36.782 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.780+0000 7f3c7e7fc640 1 -- 172.21.15.66:0/3742354796 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c800b2a30 con 0x7f3c800a98f0 2024-03-25T18:13:36.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.792+0000 7f3c7e7fc640 1 -- 172.21.15.66:0/3742354796 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2130051594 0 0) 0x7f3c800b2a30 con 0x7f3c800a98f0 2024-03-25T18:13:36.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.792+0000 7f3c7e7fc640 1 -- 172.21.15.66:0/3742354796 >> v1:172.21.15.66:6790/0 conn(0x7f3c8013f710 legacy=0x7f3c801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:36.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.792+0000 7f3c7e7fc640 1 -- 172.21.15.66:0/3742354796 >> v1:172.21.15.193:6789/0 conn(0x7f3c801430c0 legacy=0x7f3c8014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:36.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.792+0000 7f3c7e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:13:36.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.792+0000 7f3c7e7fc640 1 monclient: found mon.noname-a 2024-03-25T18:13:36.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.792+0000 7f3c7e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:36.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.792+0000 7f3c7e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.66:6789/0 2024-03-25T18:13:36.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.792+0000 7f3c7e7fc640 1 -- 172.21.15.66:0/3742354796 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c800b44c0 con 0x7f3c800a98f0 2024-03-25T18:13:36.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.792+0000 7f3c7e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:13:36.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.792+0000 7f3c7e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:13:36.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.792+0000 7f3c7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:36.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.792+0000 7f3c8606a640 5 monclient: authenticate success, global_id 4662 2024-03-25T18:13:36.795 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.792+0000 7f3c8606a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:36.795 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.792+0000 7f3c7e7fc640 1 -- 172.21.15.66:0/3742354796 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f3c700014f0 con 0x7f3c800a98f0 2024-03-25T18:13:36.795 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.792+0000 7f3c7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:36.795 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.792+0000 7f3c7e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:36.795 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.792+0000 7f3c7e7fc640 10 monclient: dump: 2024-03-25T18:13:36.795 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:36.795 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:36.795 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:36.795 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:36.795 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:36.795 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:36.795 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:36.795 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:36.795 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:36.796 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:36.796 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.792+0000 7f3c8606a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:36.810 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.808+0000 7f3c7e7fc640 1 -- 172.21.15.66:0/3742354796 <== mon.0 v1:172.21.15.66:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c700017c0 con 0x7f3c800a98f0 2024-03-25T18:13:36.810 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.808+0000 7f3c7e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:36.810 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.808+0000 7f3c7e7fc640 1 -- 172.21.15.66:0/3742354796 <== mon.0 v1:172.21.15.66:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f3c70001b50 con 0x7f3c800a98f0 2024-03-25T18:13:36.810 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.808+0000 7f3c8606a640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:36.810 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.808+0000 7f3c8606a640 4 set_mon_vals no callback set 2024-03-25T18:13:36.810 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.808+0000 7f3c7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:36.810 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.808+0000 7f3c7e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:36.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.808+0000 7f3c7e7fc640 10 monclient: dump: 2024-03-25T18:13:36.811 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:36.811 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:36.811 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:36.811 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:36.811 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:36.811 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:36.811 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:36.811 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:36.811 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:36.811 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:36.813 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.809+0000 7f3c8606a640 10 monclient: shutdown 2024-03-25T18:13:36.813 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.809+0000 7f3c8606a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:36.813 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.809+0000 7f3c8606a640 1 -- 172.21.15.66:0/3742354796 >> v1:172.21.15.66:6789/0 conn(0x7f3c800a98f0 legacy=0x7f3c800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:36.814 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.809+0000 7f3c8606a640 1 -- 172.21.15.66:0/3742354796 shutdown_connections 2024-03-25T18:13:36.814 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.809+0000 7f3c8606a640 1 -- 172.21.15.66:0/3742354796 >> 172.21.15.66:0/3742354796 conn(0x7f3c800b7ea0 msgr2=0x7f3c800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:36.814 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.809+0000 7f3c8606a640 1 -- 172.21.15.66:0/3742354796 shutdown_connections 2024-03-25T18:13:36.814 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.810+0000 7f3c8606a640 1 -- 172.21.15.66:0/3742354796 wait complete. 2024-03-25T18:13:36.814 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.810+0000 7f3c8606a640 10 monclient: build_initial_monmap 2024-03-25T18:13:36.814 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.810+0000 7f3c8606a640 10 monclient: monmap: 2024-03-25T18:13:36.814 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:13:36.814 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:36.814 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:13:36.814 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:13:36.814 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:36.814 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:13:36.814 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:13:36.815 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:13:36.815 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:13:36.815 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:36.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.811+0000 7f3c8606a640 1 librados: starting msgr at 2024-03-25T18:13:36.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.811+0000 7f3c8606a640 1 librados: starting objecter 2024-03-25T18:13:36.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.811+0000 7f3c8606a640 1 Processor -- start 2024-03-25T18:13:36.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.811+0000 7f3c8606a640 1 -- start start 2024-03-25T18:13:36.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.811+0000 7f3c8606a640 1 librados: setting wanted keys 2024-03-25T18:13:36.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.811+0000 7f3c8606a640 1 librados: calling monclient init 2024-03-25T18:13:36.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.811+0000 7f3c8606a640 10 monclient: init 2024-03-25T18:13:36.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.812+0000 7f3c8606a640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:36.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.812+0000 7f3c8606a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:13:36.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.812+0000 7f3c8606a640 10 monclient(hunting): picked mon.noname-b con 0x7f3c800a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:13:36.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.812+0000 7f3c8606a640 10 monclient(hunting): picked mon.noname-c con 0x7f3c8013f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:13:36.816 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.812+0000 7f3c8606a640 10 monclient(hunting): picked mon.noname-a con 0x7f3c801430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:13:36.816 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.812+0000 7f3c8606a640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c800b7cd0 con 0x7f3c801430c0 2024-03-25T18:13:36.816 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.812+0000 7f3c8606a640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c8013afc0 con 0x7f3c800a98f0 2024-03-25T18:13:36.816 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.812+0000 7f3c8606a640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c8013b2a0 con 0x7f3c8013f710 2024-03-25T18:13:36.816 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.812+0000 7f3c8606a640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:36.816 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.812+0000 7f3c8606a640 10 monclient(hunting): authenticate will time out at 7565.872503s 2024-03-25T18:13:36.831 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.830+0000 7f3c7ffff640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f3c801430c0 0x7f3c801e2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:53194/0 (socket says 172.21.15.66:53194) 2024-03-25T18:13:36.832 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.830+0000 7f3c7ffff640 1 -- 172.21.15.66:0/237233787 learned_addr learned my addr 172.21.15.66:0/237233787 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:13:36.834 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.832+0000 7f3c67fff640 10 client.?.objecter ms_handle_connect 0x7f3c800a98f0 2024-03-25T18:13:36.834 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.832+0000 7f3c67fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:36.835 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.833+0000 7f3c67fff640 10 client.?.objecter ms_handle_connect 0x7f3c801430c0 2024-03-25T18:13:36.835 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.833+0000 7f3c67fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:36.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.835+0000 7f3c67fff640 1 -- 172.21.15.66:0/237233787 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 20661271 0 0) 0x7f3c8013afc0 con 0x7f3c800a98f0 2024-03-25T18:13:36.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.835+0000 7f3c67fff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:36.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.835+0000 7f3c67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:36.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.835+0000 7f3c67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:36.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.835+0000 7f3c67fff640 10 monclient(hunting): my global_id is 4660 2024-03-25T18:13:36.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.835+0000 7f3c67fff640 1 -- 172.21.15.66:0/237233787 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c6c004eb0 con 0x7f3c800a98f0 2024-03-25T18:13:36.844 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.842+0000 7f3c67fff640 10 client.?.objecter ms_handle_connect 0x7f3c8013f710 2024-03-25T18:13:36.844 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.842+0000 7f3c67fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:36.848 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.846+0000 7f3c67fff640 1 -- 172.21.15.66:0/237233787 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 645562449 0 0) 0x7f3c6c004eb0 con 0x7f3c800a98f0 2024-03-25T18:13:36.848 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.846+0000 7f3c67fff640 1 -- 172.21.15.66:0/237233787 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c8013afc0 con 0x7f3c800a98f0 2024-03-25T18:13:36.848 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.846+0000 7f3c67fff640 1 -- 172.21.15.66:0/237233787 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f3c700015b0 con 0x7f3c800a98f0 2024-03-25T18:13:36.848 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.846+0000 7f3c67fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:36.848 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.846+0000 7f3c67fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:36.848 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.846+0000 7f3c67fff640 10 monclient(hunting): dump: 2024-03-25T18:13:36.848 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:36.848 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:36.848 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:36.849 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:36.849 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:36.849 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:36.849 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:36.849 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:36.849 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:36.849 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:36.849 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.846+0000 7f3c67fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:36.854 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.852+0000 7f3c67fff640 1 -- 172.21.15.66:0/237233787 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3562805050 0 0) 0x7f3c8013afc0 con 0x7f3c800a98f0 2024-03-25T18:13:36.854 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.852+0000 7f3c67fff640 1 -- 172.21.15.66:0/237233787 >> v1:172.21.15.66:6790/0 conn(0x7f3c8013f710 legacy=0x7f3c800b75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:36.854 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.852+0000 7f3c67fff640 1 -- 172.21.15.66:0/237233787 >> v1:172.21.15.66:6789/0 conn(0x7f3c801430c0 legacy=0x7f3c801e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:36.854 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.852+0000 7f3c67fff640 10 monclient: _finish_hunting 0 2024-03-25T18:13:36.854 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.852+0000 7f3c67fff640 1 monclient: found mon.b 2024-03-25T18:13:36.854 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.852+0000 7f3c67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:36.854 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.852+0000 7f3c67fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:13:36.854 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.852+0000 7f3c67fff640 1 -- 172.21.15.66:0/237233787 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c80139d20 con 0x7f3c800a98f0 2024-03-25T18:13:36.854 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.852+0000 7f3c67fff640 10 monclient: _finish_auth 0 2024-03-25T18:13:36.855 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.852+0000 7f3c67fff640 10 monclient: _check_auth_tickets 2024-03-25T18:13:36.855 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.852+0000 7f3c67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:36.855 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.852+0000 7f3c8606a640 5 monclient: authenticate success, global_id 4660 2024-03-25T18:13:36.855 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.852+0000 7f3c8606a640 10 monclient: _renew_subs 2024-03-25T18:13:36.855 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.852+0000 7f3c8606a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:13:36.855 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.852+0000 7f3c8606a640 1 -- 172.21.15.66:0/237233787 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c801397f0 con 0x7f3c800a98f0 2024-03-25T18:13:36.855 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.852+0000 7f3c8606a640 10 client.4660.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:36.855 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.852+0000 7f3c8606a640 10 monclient: _renew_subs 2024-03-25T18:13:36.855 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.852+0000 7f3c8606a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:13:36.855 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.852+0000 7f3c8606a640 1 -- 172.21.15.66:0/237233787 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c8013bb80 con 0x7f3c800a98f0 2024-03-25T18:13:36.855 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.852+0000 7f3c8606a640 1 librados: init done 2024-03-25T18:13:36.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.853+0000 7f3c8606a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:36.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.853+0000 7f3c8606a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:36.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.853+0000 7f3c8606a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:13:36.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.853+0000 7f3c8606a640 1 -- 172.21.15.66:0/237233787 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c8014f2d0 con 0x7f3c800a98f0 2024-03-25T18:13:36.876 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.874+0000 7f3c67fff640 1 -- 172.21.15.66:0/237233787 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c700038a0 con 0x7f3c800a98f0 2024-03-25T18:13:36.876 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.874+0000 7f3c67fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:36.876 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.874+0000 7f3c67fff640 1 -- 172.21.15.66:0/237233787 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f3c70003cf0 con 0x7f3c800a98f0 2024-03-25T18:13:36.876 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.874+0000 7f3c67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:36.876 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.874+0000 7f3c67fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:36.876 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.874+0000 7f3c67fff640 10 monclient: dump: 2024-03-25T18:13:36.876 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:36.876 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:36.877 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:36.877 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:36.877 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:36.877 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:36.877 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:36.877 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:36.877 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:36.877 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:36.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.874+0000 7f3c7dffb640 4 set_mon_vals no callback set 2024-03-25T18:13:36.878 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.875+0000 7f3c67fff640 1 -- 172.21.15.66:0/237233787 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f3c70003f10 con 0x7f3c800a98f0 2024-03-25T18:13:36.878 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.876+0000 7f3c7effd640 10 monclient: get_auth_request con 0x7f3c6c06dce0 auth_method 0 2024-03-25T18:13:36.878 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.876+0000 7f3c67fff640 10 client.4660.objecter ms_handle_connect 0x7f3c6c06dce0 2024-03-25T18:13:36.880 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.878+0000 7f3c67fff640 1 -- 172.21.15.66:0/237233787 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f3c700537b0 con 0x7f3c800a98f0 2024-03-25T18:13:36.880 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.878+0000 7f3c67fff640 10 client.4660.objecter ms_dispatch 0x7f3c80000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:36.880 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.878+0000 7f3c67fff640 3 client.4660.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:36.880 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.878+0000 7f3c67fff640 3 client.4660.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:36.880 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:36.878+0000 7f3c67fff640 20 client.4660.objecter dump_active .. 0 homeless 2024-03-25T18:13:37.140 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:37.138+0000 7f3c67fff640 1 -- 172.21.15.66:0/237233787 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3c7004f2c0 con 0x7f3c800a98f0 2024-03-25T18:13:37.140 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:37.138+0000 7f3c67fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:37.140 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:37.138+0000 7f3c67fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:37.397 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:37.395+0000 7f3c8606a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:37.397 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:37.395+0000 7f3c8606a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:37.397 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:37.395+0000 7f3c8606a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:13:37.397 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:37.395+0000 7f3c8606a640 1 -- 172.21.15.66:0/237233787 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c800627b0 con 0x7f3c800a98f0 2024-03-25T18:13:37.414 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:37.413+0000 7f3c67fff640 1 -- 172.21.15.66:0/237233787 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f3c700511f0 con 0x7f3c800a98f0 2024-03-25T18:13:37.414 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:37.413+0000 7f3c67fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:37.414 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:37.413+0000 7f3c67fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:37.415 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:13:37.415 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:37.417 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:37.415+0000 7f3c8606a640 10 librados: watch_flush enter 2024-03-25T18:13:37.417 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:37.415+0000 7f3c8606a640 10 librados: watch_flush exit 2024-03-25T18:13:37.417 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:37.415+0000 7f3c8606a640 20 client.4660.objecter shutdown clearing up homeless session... 2024-03-25T18:13:37.418 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:37.415+0000 7f3c8606a640 10 client.4660.objecter successfully canceled tick 2024-03-25T18:13:37.418 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:37.415+0000 7f3c8606a640 1 -- 172.21.15.66:0/237233787 >> v1:172.21.15.193:6800/2275676803 conn(0x7f3c6c06dce0 legacy=0x7f3c6c0701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:37.418 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:37.415+0000 7f3c8606a640 10 monclient: shutdown 2024-03-25T18:13:37.418 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:37.415+0000 7f3c8606a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:37.418 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:37.415+0000 7f3c8606a640 1 -- 172.21.15.66:0/237233787 >> v1:172.21.15.193:6789/0 conn(0x7f3c800a98f0 legacy=0x7f3c8012ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:37.418 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:37.415+0000 7f3c8606a640 1 -- 172.21.15.66:0/237233787 shutdown_connections 2024-03-25T18:13:37.418 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:37.415+0000 7f3c8606a640 1 -- 172.21.15.66:0/237233787 >> 172.21.15.66:0/237233787 conn(0x7f3c800b7ea0 msgr2=0x7f3c80074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:37.418 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:37.415+0000 7f3c8606a640 1 -- 172.21.15.66:0/237233787 shutdown_connections 2024-03-25T18:13:37.418 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:37.415+0000 7f3c8606a640 1 -- 172.21.15.66:0/237233787 wait complete. 2024-03-25T18:13:37.418 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:37.415+0000 7f3c8606a640 1 librados: shutdown 2024-03-25T18:13:37.428 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:44.235 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:44.355 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.354+0000 7f520764e640 10 monclient: get_monmap_and_config 2024-03-25T18:13:44.356 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.354+0000 7f520764e640 10 monclient: build_initial_monmap 2024-03-25T18:13:44.356 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.354+0000 7f520764e640 10 monclient: monmap: 2024-03-25T18:13:44.356 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:13:44.356 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:44.356 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:13:44.355843+0000 2024-03-25T18:13:44.356 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:13:44.355843+0000 2024-03-25T18:13:44.356 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:44.356 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:44.356 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:13:44.356 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:13:44.356 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:13:44.356 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:44.357 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.355+0000 7f520764e640 1 Processor -- start 2024-03-25T18:13:44.357 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.355+0000 7f520764e640 1 -- start start 2024-03-25T18:13:44.357 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.355+0000 7f520764e640 10 monclient: init 2024-03-25T18:13:44.358 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.356+0000 7f520764e640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:44.359 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.356+0000 7f520764e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:13:44.359 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.356+0000 7f520764e640 10 monclient(hunting): picked mon.noname-c con 0x7f52000705b0 addr v1:172.21.15.193:6789/0 2024-03-25T18:13:44.359 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.356+0000 7f520764e640 10 monclient(hunting): picked mon.noname-a con 0x7f52001402d0 addr v1:172.21.15.66:6789/0 2024-03-25T18:13:44.359 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.356+0000 7f520764e640 10 monclient(hunting): picked mon.noname-b con 0x7f5200149ae0 addr v1:172.21.15.66:6790/0 2024-03-25T18:13:44.359 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.356+0000 7f520764e640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52000718c0 con 0x7f52001402d0 2024-03-25T18:13:44.359 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.356+0000 7f520764e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52000714c0 con 0x7f52000705b0 2024-03-25T18:13:44.359 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.356+0000 7f520764e640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5200072b50 con 0x7f5200149ae0 2024-03-25T18:13:44.359 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.356+0000 7f520764e640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:44.359 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.356+0000 7f520764e640 10 monclient(hunting): authenticate will time out at 7573.416803s 2024-03-25T18:13:44.370 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.368+0000 7f52053ea640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f52000705b0 0x7f520013d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:49090/0 (socket says 172.21.15.66:49090) 2024-03-25T18:13:44.370 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.368+0000 7f52053ea640 1 -- 172.21.15.66:0/520683267 learned_addr learned my addr 172.21.15.66:0/520683267 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:13:44.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.378+0000 7f51f7fff640 1 -- 172.21.15.66:0/520683267 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484221597 0 0) 0x7f52000718c0 con 0x7f52001402d0 2024-03-25T18:13:44.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.378+0000 7f51f7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:44.379 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.378+0000 7f51f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:44.380 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.378+0000 7f51f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:44.380 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.378+0000 7f51f7fff640 10 monclient(hunting): my global_id is 4668 2024-03-25T18:13:44.380 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.378+0000 7f51f7fff640 1 -- 172.21.15.66:0/520683267 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51ec003520 con 0x7f52001402d0 2024-03-25T18:13:44.380 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.378+0000 7f51f7fff640 1 -- 172.21.15.66:0/520683267 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3853559946 0 0) 0x7f52000714c0 con 0x7f52000705b0 2024-03-25T18:13:44.380 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.378+0000 7f51f7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:44.380 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.378+0000 7f51f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:44.380 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.378+0000 7f51f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:44.380 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.378+0000 7f51f7fff640 10 monclient(hunting): my global_id is 4663 2024-03-25T18:13:44.381 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.378+0000 7f51f7fff640 1 -- 172.21.15.66:0/520683267 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52000718c0 con 0x7f52000705b0 2024-03-25T18:13:44.382 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.380+0000 7f51f7fff640 1 -- 172.21.15.66:0/520683267 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3823930543 0 0) 0x7f5200072b50 con 0x7f5200149ae0 2024-03-25T18:13:44.382 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.380+0000 7f51f7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:44.382 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.380+0000 7f51f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:44.382 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.380+0000 7f51f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:44.382 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.380+0000 7f51f7fff640 10 monclient(hunting): my global_id is 4664 2024-03-25T18:13:44.382 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.380+0000 7f51f7fff640 1 -- 172.21.15.66:0/520683267 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52000714c0 con 0x7f5200149ae0 2024-03-25T18:13:44.394 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.392+0000 7f51f7fff640 1 -- 172.21.15.66:0/520683267 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2138992478 0 0) 0x7f52000714c0 con 0x7f5200149ae0 2024-03-25T18:13:44.394 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.392+0000 7f51f7fff640 1 -- 172.21.15.66:0/520683267 >> v1:172.21.15.193:6789/0 conn(0x7f52000705b0 legacy=0x7f520013d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:44.394 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.392+0000 7f51f7fff640 1 -- 172.21.15.66:0/520683267 >> v1:172.21.15.66:6789/0 conn(0x7f52001402d0 legacy=0x7f5200140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:44.394 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.392+0000 7f51f7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:13:44.394 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.392+0000 7f51f7fff640 1 monclient: found mon.noname-b 2024-03-25T18:13:44.394 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.392+0000 7f51f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:44.394 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.392+0000 7f51f7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.66:6790/0 2024-03-25T18:13:44.394 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.392+0000 7f51f7fff640 1 -- 172.21.15.66:0/520683267 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f520013dd10 con 0x7f5200149ae0 2024-03-25T18:13:44.394 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.392+0000 7f51f7fff640 10 monclient: _finish_auth 0 2024-03-25T18:13:44.394 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.392+0000 7f51f7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:13:44.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.392+0000 7f51f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:44.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.392+0000 7f51f7fff640 1 -- 172.21.15.66:0/520683267 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f51f8001930 con 0x7f5200149ae0 2024-03-25T18:13:44.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.392+0000 7f520764e640 5 monclient: authenticate success, global_id 4664 2024-03-25T18:13:44.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.392+0000 7f520764e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:44.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.392+0000 7f51f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:44.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.392+0000 7f51f7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:44.395 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.392+0000 7f51f7fff640 10 monclient: dump: 2024-03-25T18:13:44.395 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:44.395 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:44.395 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:44.395 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:44.395 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:44.395 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:44.395 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:44.395 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:44.396 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:44.396 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:44.396 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.392+0000 7f520764e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:44.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.402+0000 7f51f7fff640 1 -- 172.21.15.66:0/520683267 <== mon.2 v1:172.21.15.66:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51f8001a90 con 0x7f5200149ae0 2024-03-25T18:13:44.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.402+0000 7f51f7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:44.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.402+0000 7f51f7fff640 1 -- 172.21.15.66:0/520683267 <== mon.2 v1:172.21.15.66:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f51f8001ee0 con 0x7f5200149ae0 2024-03-25T18:13:44.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.402+0000 7f51f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:44.403 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.402+0000 7f51f7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:44.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.402+0000 7f51f7fff640 10 monclient: dump: 2024-03-25T18:13:44.404 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:44.404 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:44.404 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:44.404 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:44.404 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:44.404 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:44.404 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:44.404 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:44.404 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:44.404 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:44.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.402+0000 7f520764e640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:44.404 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.402+0000 7f520764e640 4 set_mon_vals no callback set 2024-03-25T18:13:44.406 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.402+0000 7f520764e640 10 monclient: shutdown 2024-03-25T18:13:44.406 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.403+0000 7f520764e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:44.406 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.403+0000 7f520764e640 1 -- 172.21.15.66:0/520683267 >> v1:172.21.15.66:6790/0 conn(0x7f5200149ae0 legacy=0x7f520014bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:44.406 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.403+0000 7f520764e640 1 -- 172.21.15.66:0/520683267 shutdown_connections 2024-03-25T18:13:44.406 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.403+0000 7f520764e640 1 -- 172.21.15.66:0/520683267 >> 172.21.15.66:0/520683267 conn(0x7f52000b5d10 msgr2=0x7f52000b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:44.406 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.403+0000 7f520764e640 1 -- 172.21.15.66:0/520683267 shutdown_connections 2024-03-25T18:13:44.406 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.403+0000 7f520764e640 1 -- 172.21.15.66:0/520683267 wait complete. 2024-03-25T18:13:44.406 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.404+0000 7f520764e640 10 monclient: build_initial_monmap 2024-03-25T18:13:44.406 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.404+0000 7f520764e640 10 monclient: monmap: 2024-03-25T18:13:44.407 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:13:44.407 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:44.407 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:13:44.407 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:13:44.407 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:44.407 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:13:44.407 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:13:44.407 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:13:44.407 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:13:44.407 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:44.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.404+0000 7f520764e640 1 librados: starting msgr at 2024-03-25T18:13:44.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.404+0000 7f520764e640 1 librados: starting objecter 2024-03-25T18:13:44.408 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.404+0000 7f520764e640 1 Processor -- start 2024-03-25T18:13:44.408 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.404+0000 7f520764e640 1 -- start start 2024-03-25T18:13:44.408 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.404+0000 7f520764e640 1 librados: setting wanted keys 2024-03-25T18:13:44.408 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.404+0000 7f520764e640 1 librados: calling monclient init 2024-03-25T18:13:44.408 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.404+0000 7f520764e640 10 monclient: init 2024-03-25T18:13:44.409 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.405+0000 7f520764e640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:44.409 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.405+0000 7f520764e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:44.409 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.405+0000 7f520764e640 10 monclient(hunting): picked mon.noname-c con 0x7f52000705b0 addr v1:172.21.15.66:6790/0 2024-03-25T18:13:44.409 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.405+0000 7f520764e640 10 monclient(hunting): picked mon.noname-b con 0x7f52001402d0 addr v1:172.21.15.193:6789/0 2024-03-25T18:13:44.409 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.405+0000 7f520764e640 10 monclient(hunting): picked mon.noname-a con 0x7f5200149ae0 addr v1:172.21.15.66:6789/0 2024-03-25T18:13:44.409 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.405+0000 7f520764e640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5200141ee0 con 0x7f5200149ae0 2024-03-25T18:13:44.409 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.405+0000 7f520764e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52001d7db0 con 0x7f52001402d0 2024-03-25T18:13:44.409 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.405+0000 7f520764e640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52001d82d0 con 0x7f52000705b0 2024-03-25T18:13:44.409 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.405+0000 7f520764e640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:44.409 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.405+0000 7f520764e640 10 monclient(hunting): authenticate will time out at 7573.465976s 2024-03-25T18:13:44.421 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.420+0000 7f52053ea640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f52000705b0 0x7f52001d74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:56510/0 (socket says 172.21.15.66:56510) 2024-03-25T18:13:44.421 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.420+0000 7f52053ea640 1 -- 172.21.15.66:0/979857158 learned_addr learned my addr 172.21.15.66:0/979857158 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:13:44.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.421+0000 7f51f5ffb640 10 client.?.objecter ms_handle_connect 0x7f52001402d0 2024-03-25T18:13:44.422 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.421+0000 7f51f5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:44.427 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.426+0000 7f51f5ffb640 1 -- 172.21.15.66:0/979857158 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398943927 0 0) 0x7f52001d7db0 con 0x7f52001402d0 2024-03-25T18:13:44.428 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.426+0000 7f51f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:44.428 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.426+0000 7f51f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:44.428 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.426+0000 7f51f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:44.428 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.426+0000 7f51f5ffb640 10 monclient(hunting): my global_id is 4666 2024-03-25T18:13:44.428 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.426+0000 7f51f5ffb640 1 -- 172.21.15.66:0/979857158 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51e4004eb0 con 0x7f52001402d0 2024-03-25T18:13:44.428 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.427+0000 7f51f5ffb640 10 client.?.objecter ms_handle_connect 0x7f52000705b0 2024-03-25T18:13:44.429 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.427+0000 7f51f5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:44.434 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.432+0000 7f51f5ffb640 10 client.?.objecter ms_handle_connect 0x7f5200149ae0 2024-03-25T18:13:44.434 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.432+0000 7f51f5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:44.434 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.432+0000 7f51f5ffb640 1 -- 172.21.15.66:0/979857158 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1574352864 0 0) 0x7f51e4004eb0 con 0x7f52001402d0 2024-03-25T18:13:44.434 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.432+0000 7f51f5ffb640 1 -- 172.21.15.66:0/979857158 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52001d7db0 con 0x7f52001402d0 2024-03-25T18:13:44.434 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.432+0000 7f51f5ffb640 1 -- 172.21.15.66:0/979857158 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f51f0001950 con 0x7f52001402d0 2024-03-25T18:13:44.434 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.432+0000 7f51f5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:44.434 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.432+0000 7f51f5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:44.434 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.432+0000 7f51f5ffb640 10 monclient(hunting): dump: 2024-03-25T18:13:44.434 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:44.434 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:44.434 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:44.435 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:44.435 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:44.435 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:44.435 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:44.435 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:44.435 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:44.435 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:44.435 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.432+0000 7f51f5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:44.440 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.439+0000 7f51f5ffb640 1 -- 172.21.15.66:0/979857158 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1301766579 0 0) 0x7f5200141ee0 con 0x7f5200149ae0 2024-03-25T18:13:44.441 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.439+0000 7f51f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:44.441 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.439+0000 7f51f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:44.441 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.439+0000 7f51f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:44.441 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.439+0000 7f51f5ffb640 10 monclient(hunting): my global_id is 4671 2024-03-25T18:13:44.441 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.439+0000 7f51f5ffb640 1 -- 172.21.15.66:0/979857158 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51e4004eb0 con 0x7f5200149ae0 2024-03-25T18:13:44.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.445+0000 7f51f5ffb640 1 -- 172.21.15.66:0/979857158 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2498397389 0 0) 0x7f52001d82d0 con 0x7f52000705b0 2024-03-25T18:13:44.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.446+0000 7f51f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:44.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.446+0000 7f51f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:44.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.446+0000 7f51f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:44.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.446+0000 7f51f5ffb640 10 monclient(hunting): my global_id is 4667 2024-03-25T18:13:44.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.446+0000 7f51f5ffb640 1 -- 172.21.15.66:0/979857158 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5200141ee0 con 0x7f52000705b0 2024-03-25T18:13:44.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.452+0000 7f51f5ffb640 1 -- 172.21.15.66:0/979857158 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2195326395 0 0) 0x7f52001d7db0 con 0x7f52001402d0 2024-03-25T18:13:44.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.452+0000 7f51f5ffb640 1 -- 172.21.15.66:0/979857158 >> v1:172.21.15.66:6790/0 conn(0x7f52000705b0 legacy=0x7f52001d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:44.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.452+0000 7f51f5ffb640 1 -- 172.21.15.66:0/979857158 >> v1:172.21.15.66:6789/0 conn(0x7f5200149ae0 legacy=0x7f52001e1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:44.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.452+0000 7f51f5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:13:44.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.452+0000 7f51f5ffb640 1 monclient: found mon.b 2024-03-25T18:13:44.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.452+0000 7f51f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:44.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.452+0000 7f51f5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:13:44.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.452+0000 7f51f5ffb640 1 -- 172.21.15.66:0/979857158 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f520006cbf0 con 0x7f52001402d0 2024-03-25T18:13:44.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.452+0000 7f51f5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:13:44.455 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.452+0000 7f51f5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:13:44.455 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.452+0000 7f51f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:44.455 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.452+0000 7f520764e640 5 monclient: authenticate success, global_id 4666 2024-03-25T18:13:44.455 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.452+0000 7f520764e640 10 monclient: _renew_subs 2024-03-25T18:13:44.455 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.452+0000 7f520764e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:13:44.455 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.452+0000 7f520764e640 1 -- 172.21.15.66:0/979857158 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f520006ce40 con 0x7f52001402d0 2024-03-25T18:13:44.455 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.452+0000 7f520764e640 10 client.4666.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:44.455 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.452+0000 7f520764e640 10 monclient: _renew_subs 2024-03-25T18:13:44.455 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.452+0000 7f520764e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:13:44.455 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.452+0000 7f520764e640 1 -- 172.21.15.66:0/979857158 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f52001490c0 con 0x7f52001402d0 2024-03-25T18:13:44.455 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.452+0000 7f520764e640 1 librados: init done 2024-03-25T18:13:44.456 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.453+0000 7f520764e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:44.456 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.453+0000 7f520764e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:44.456 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.453+0000 7f520764e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:13:44.456 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.453+0000 7f520764e640 1 -- 172.21.15.66:0/979857158 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52001d7f40 con 0x7f52001402d0 2024-03-25T18:13:44.740 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.738+0000 7f51f5ffb640 1 -- 172.21.15.66:0/979857158 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51f0003610 con 0x7f52001402d0 2024-03-25T18:13:44.740 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.738+0000 7f51f5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:44.740 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.738+0000 7f51f5ffb640 1 -- 172.21.15.66:0/979857158 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f51f0003920 con 0x7f52001402d0 2024-03-25T18:13:44.740 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.738+0000 7f51f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:44.740 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.738+0000 7f51f5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:44.740 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.738+0000 7f51f5ffb640 10 monclient: dump: 2024-03-25T18:13:44.741 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:44.741 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:44.741 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:44.741 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:44.741 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:44.741 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:44.741 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:44.741 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:44.741 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:44.741 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:44.741 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.738+0000 7f51f77fe640 4 set_mon_vals no callback set 2024-03-25T18:13:44.742 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.740+0000 7f51f5ffb640 1 -- 172.21.15.66:0/979857158 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f51f00198f0 con 0x7f52001402d0 2024-03-25T18:13:44.742 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.740+0000 7f52053ea640 10 monclient: get_auth_request con 0x7f51e406de10 auth_method 0 2024-03-25T18:13:44.742 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.741+0000 7f51f5ffb640 10 client.4666.objecter ms_handle_connect 0x7f51e406de10 2024-03-25T18:13:44.745 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.744+0000 7f51f5ffb640 1 -- 172.21.15.66:0/979857158 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1504299344 0 0) 0x7f51f0053640 con 0x7f52001402d0 2024-03-25T18:13:44.745 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.744+0000 7f51f5ffb640 10 client.4666.objecter ms_dispatch 0x7f5200000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:44.745 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.744+0000 7f51f5ffb640 3 client.4666.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:44.745 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.744+0000 7f51f5ffb640 3 client.4666.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:44.746 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.744+0000 7f51f5ffb640 20 client.4666.objecter dump_active .. 0 homeless 2024-03-25T18:13:44.747 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.745+0000 7f51f5ffb640 1 -- 172.21.15.66:0/979857158 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f51f004ef80 con 0x7f52001402d0 2024-03-25T18:13:44.747 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.746+0000 7f51f5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:44.747 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.746+0000 7f51f5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:44.987 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.986+0000 7f520764e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:44.987 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.986+0000 7f520764e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:44.987 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.986+0000 7f520764e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:13:44.987 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:44.986+0000 7f520764e640 1 -- 172.21.15.66:0/979857158 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52001d7bb0 con 0x7f52001402d0 2024-03-25T18:13:45.054 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:45.053+0000 7f51f5ffb640 1 -- 172.21.15.66:0/979857158 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1603418227) 0x7f51f0051080 con 0x7f52001402d0 2024-03-25T18:13:45.054 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:45.053+0000 7f51f5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:45.055 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:45.053+0000 7f51f5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:45.055 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:13:45.055 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":10,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:03:52.380875+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:03:52.380875+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:45.057 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:45.055+0000 7f520764e640 10 librados: watch_flush enter 2024-03-25T18:13:45.057 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:45.055+0000 7f520764e640 10 librados: watch_flush exit 2024-03-25T18:13:45.058 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:45.055+0000 7f520764e640 20 client.4666.objecter shutdown clearing up homeless session... 2024-03-25T18:13:45.058 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:45.055+0000 7f520764e640 10 client.4666.objecter successfully canceled tick 2024-03-25T18:13:45.058 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:45.056+0000 7f520764e640 1 -- 172.21.15.66:0/979857158 >> v1:172.21.15.193:6800/2275676803 conn(0x7f51e406de10 legacy=0x7f51e40702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:45.058 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:45.056+0000 7f520764e640 10 monclient: shutdown 2024-03-25T18:13:45.058 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:45.056+0000 7f520764e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:45.058 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:45.056+0000 7f520764e640 1 -- 172.21.15.66:0/979857158 >> v1:172.21.15.193:6789/0 conn(0x7f52001402d0 legacy=0x7f520006c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:45.058 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:45.056+0000 7f520764e640 1 -- 172.21.15.66:0/979857158 shutdown_connections 2024-03-25T18:13:45.058 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:45.056+0000 7f520764e640 1 -- 172.21.15.66:0/979857158 >> 172.21.15.66:0/979857158 conn(0x7f52000b5d10 msgr2=0x7f52000b0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:45.058 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:45.056+0000 7f520764e640 1 -- 172.21.15.66:0/979857158 shutdown_connections 2024-03-25T18:13:45.058 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:45.056+0000 7f520764e640 1 -- 172.21.15.66:0/979857158 wait complete. 2024-03-25T18:13:45.058 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:45.056+0000 7f520764e640 1 librados: shutdown 2024-03-25T18:13:45.069 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:50.586 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:10:21.566 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:50.639 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:10:21.620 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:50.681 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:10:21.661 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:50.691 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:10:21.649 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:50.694 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:10:21.652 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:50.701 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:10:21.681 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:50.716 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:10:21.697 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:50.719 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:10:21.677 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:50.741 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:10:21.721 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:51.875 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:51.974 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:51.974+0000 7f7db2f4b640 10 monclient: get_monmap_and_config 2024-03-25T18:13:51.974 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:51.974+0000 7f7db2f4b640 10 monclient: build_initial_monmap 2024-03-25T18:13:51.975 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:51.974+0000 7f7db2f4b640 10 monclient: monmap: 2024-03-25T18:13:51.975 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:13:51.975 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:51.975 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:13:51.974594+0000 2024-03-25T18:13:51.975 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:13:51.974594+0000 2024-03-25T18:13:51.975 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:51.975 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:51.975 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:13:51.975 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:13:51.975 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:13:51.975 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:51.976 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:51.974+0000 7f7db2f4b640 1 Processor -- start 2024-03-25T18:13:51.977 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:51.974+0000 7f7db2f4b640 1 -- start start 2024-03-25T18:13:51.977 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:51.974+0000 7f7db2f4b640 10 monclient: init 2024-03-25T18:13:51.977 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:51.975+0000 7f7db2f4b640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:51.977 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:51.975+0000 7f7db2f4b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:13:51.977 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:51.975+0000 7f7db2f4b640 10 monclient(hunting): picked mon.noname-c con 0x7f7dac0b4d60 addr v1:172.21.15.193:6789/0 2024-03-25T18:13:51.977 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:51.975+0000 7f7db2f4b640 10 monclient(hunting): picked mon.noname-a con 0x7f7dac0b1730 addr v1:172.21.15.66:6789/0 2024-03-25T18:13:51.977 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:51.975+0000 7f7db2f4b640 10 monclient(hunting): picked mon.noname-b con 0x7f7dac14be90 addr v1:172.21.15.66:6790/0 2024-03-25T18:13:51.977 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:51.975+0000 7f7db2f4b640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dac0af940 con 0x7f7dac0b1730 2024-03-25T18:13:51.977 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:51.975+0000 7f7db2f4b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dac0b0a70 con 0x7f7dac0b4d60 2024-03-25T18:13:51.977 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:51.975+0000 7f7db2f4b640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dac0b0d50 con 0x7f7dac14be90 2024-03-25T18:13:51.977 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:51.975+0000 7f7db2f4b640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:51.977 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:51.975+0000 7f7db2f4b640 10 monclient(hunting): authenticate will time out at 7581.034716s 2024-03-25T18:13:51.979 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:51.979+0000 7f7db0ce7640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f7dac0b4d60 0x7f7dac13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:53074/0 (socket says 172.21.15.66:53074) 2024-03-25T18:13:51.979 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:51.979+0000 7f7db0ce7640 1 -- 172.21.15.66:0/1739671633 learned_addr learned my addr 172.21.15.66:0/1739671633 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:13:52.020 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.019+0000 7f7dab7fe640 1 -- 172.21.15.66:0/1739671633 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940868577 0 0) 0x7f7dac0b0a70 con 0x7f7dac0b4d60 2024-03-25T18:13:52.020 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.020+0000 7f7dab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:52.020 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.020+0000 7f7dab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:52.021 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.020+0000 7f7dab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:52.021 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.020+0000 7f7dab7fe640 10 monclient(hunting): my global_id is 4669 2024-03-25T18:13:52.021 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.020+0000 7f7dab7fe640 1 -- 172.21.15.66:0/1739671633 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d98003520 con 0x7f7dac0b4d60 2024-03-25T18:13:52.025 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.025+0000 7f7dab7fe640 1 -- 172.21.15.66:0/1739671633 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1339175392 0 0) 0x7f7dac0b0d50 con 0x7f7dac14be90 2024-03-25T18:13:52.025 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.025+0000 7f7dab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:52.025 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.025+0000 7f7dab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:52.025 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.025+0000 7f7dab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:52.025 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.025+0000 7f7dab7fe640 10 monclient(hunting): my global_id is 4670 2024-03-25T18:13:52.025 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.025+0000 7f7dab7fe640 1 -- 172.21.15.66:0/1739671633 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dac0b0a70 con 0x7f7dac14be90 2024-03-25T18:13:52.029 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.028+0000 7f7dab7fe640 1 -- 172.21.15.66:0/1739671633 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1899403461 0 0) 0x7f7dac0af940 con 0x7f7dac0b1730 2024-03-25T18:13:52.029 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.028+0000 7f7dab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:52.029 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.028+0000 7f7dab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:52.029 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.028+0000 7f7dab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:52.029 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.028+0000 7f7dab7fe640 10 monclient(hunting): my global_id is 4674 2024-03-25T18:13:52.029 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.028+0000 7f7dab7fe640 1 -- 172.21.15.66:0/1739671633 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dac0b0d50 con 0x7f7dac0b1730 2024-03-25T18:13:52.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.040+0000 7f7dab7fe640 1 -- 172.21.15.66:0/1739671633 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1930868199 0 0) 0x7f7d98003520 con 0x7f7dac0b4d60 2024-03-25T18:13:52.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.040+0000 7f7dab7fe640 1 -- 172.21.15.66:0/1739671633 >> v1:172.21.15.66:6790/0 conn(0x7f7dac14be90 legacy=0x7f7dac14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:52.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.040+0000 7f7dab7fe640 1 -- 172.21.15.66:0/1739671633 >> v1:172.21.15.66:6789/0 conn(0x7f7dac0b1730 legacy=0x7f7dac0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:52.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.040+0000 7f7dab7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:13:52.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.040+0000 7f7dab7fe640 1 monclient: found mon.noname-c 2024-03-25T18:13:52.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.040+0000 7f7dab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:52.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.040+0000 7f7dab7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:13:52.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.040+0000 7f7dab7fe640 1 -- 172.21.15.66:0/1739671633 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dac0b1030 con 0x7f7dac0b4d60 2024-03-25T18:13:52.041 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.040+0000 7f7dab7fe640 10 monclient: _finish_auth 0 2024-03-25T18:13:52.042 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.040+0000 7f7dab7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:13:52.042 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.040+0000 7f7dab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:52.042 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.040+0000 7f7dab7fe640 1 -- 172.21.15.66:0/1739671633 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7d94001690 con 0x7f7dac0b4d60 2024-03-25T18:13:52.042 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.040+0000 7f7dab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:52.042 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.040+0000 7f7dab7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:52.042 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.040+0000 7f7dab7fe640 10 monclient: dump: 2024-03-25T18:13:52.042 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:52.042 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:52.042 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:52.042 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:52.042 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:52.042 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:52.042 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:52.042 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:52.043 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:52.043 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:52.043 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.040+0000 7f7db2f4b640 5 monclient: authenticate success, global_id 4669 2024-03-25T18:13:52.043 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.040+0000 7f7db2f4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:52.055 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.055+0000 7f7dab7fe640 1 -- 172.21.15.66:0/1739671633 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d940017f0 con 0x7f7dac0b4d60 2024-03-25T18:13:52.056 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.055+0000 7f7dab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:52.056 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.055+0000 7f7dab7fe640 1 -- 172.21.15.66:0/1739671633 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7d94001c00 con 0x7f7dac0b4d60 2024-03-25T18:13:52.056 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.055+0000 7f7db2f4b640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:52.056 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.055+0000 7f7db2f4b640 4 set_mon_vals no callback set 2024-03-25T18:13:52.056 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.055+0000 7f7dab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:52.056 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.055+0000 7f7dab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:52.056 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.055+0000 7f7dab7fe640 10 monclient: dump: 2024-03-25T18:13:52.056 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:52.056 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:52.056 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:52.056 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:52.056 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:52.056 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:52.057 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:52.057 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:52.057 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:52.057 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:52.059 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.056+0000 7f7db2f4b640 10 monclient: shutdown 2024-03-25T18:13:52.059 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.056+0000 7f7db2f4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:52.059 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.056+0000 7f7db2f4b640 1 -- 172.21.15.66:0/1739671633 >> v1:172.21.15.193:6789/0 conn(0x7f7dac0b4d60 legacy=0x7f7dac13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:52.059 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.056+0000 7f7db2f4b640 1 -- 172.21.15.66:0/1739671633 shutdown_connections 2024-03-25T18:13:52.059 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.056+0000 7f7db2f4b640 1 -- 172.21.15.66:0/1739671633 >> 172.21.15.66:0/1739671633 conn(0x7f7dac0617a0 msgr2=0x7f7dac061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:52.059 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.056+0000 7f7db2f4b640 1 -- 172.21.15.66:0/1739671633 shutdown_connections 2024-03-25T18:13:52.059 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.056+0000 7f7db2f4b640 1 -- 172.21.15.66:0/1739671633 wait complete. 2024-03-25T18:13:52.060 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.057+0000 7f7db2f4b640 10 monclient: build_initial_monmap 2024-03-25T18:13:52.060 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.057+0000 7f7db2f4b640 10 monclient: monmap: 2024-03-25T18:13:52.060 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:13:52.060 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:52.060 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:13:52.060 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:13:52.060 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:52.060 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:13:52.060 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:13:52.060 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:13:52.060 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:13:52.060 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:52.060 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.057+0000 7f7db2f4b640 1 librados: starting msgr at 2024-03-25T18:13:52.060 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.057+0000 7f7db2f4b640 1 librados: starting objecter 2024-03-25T18:13:52.061 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.057+0000 7f7db2f4b640 1 Processor -- start 2024-03-25T18:13:52.061 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.057+0000 7f7db2f4b640 1 -- start start 2024-03-25T18:13:52.061 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.057+0000 7f7db2f4b640 1 librados: setting wanted keys 2024-03-25T18:13:52.061 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.057+0000 7f7db2f4b640 1 librados: calling monclient init 2024-03-25T18:13:52.061 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.057+0000 7f7db2f4b640 10 monclient: init 2024-03-25T18:13:52.061 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.058+0000 7f7db2f4b640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:52.061 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.058+0000 7f7db2f4b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:13:52.061 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.058+0000 7f7db2f4b640 10 monclient(hunting): picked mon.noname-b con 0x7f7dac0b1730 addr v1:172.21.15.193:6789/0 2024-03-25T18:13:52.061 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.058+0000 7f7db2f4b640 10 monclient(hunting): picked mon.noname-c con 0x7f7dac0b4d60 addr v1:172.21.15.66:6790/0 2024-03-25T18:13:52.061 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.058+0000 7f7db2f4b640 10 monclient(hunting): picked mon.noname-a con 0x7f7dac14be90 addr v1:172.21.15.66:6789/0 2024-03-25T18:13:52.061 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.058+0000 7f7db2f4b640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dac141ce0 con 0x7f7dac14be90 2024-03-25T18:13:52.061 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.058+0000 7f7db2f4b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dac142610 con 0x7f7dac0b1730 2024-03-25T18:13:52.061 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.058+0000 7f7db2f4b640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dac1dae80 con 0x7f7dac0b4d60 2024-03-25T18:13:52.061 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.058+0000 7f7db2f4b640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:52.062 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.058+0000 7f7db2f4b640 10 monclient(hunting): authenticate will time out at 7581.117926s 2024-03-25T18:13:52.062 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.062+0000 7f7db14e8640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f7dac14be90 0x7f7dac140380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:34716/0 (socket says 172.21.15.66:34716) 2024-03-25T18:13:52.062 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.062+0000 7f7db14e8640 1 -- 172.21.15.66:0/3414616585 learned_addr learned my addr 172.21.15.66:0/3414616585 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:13:52.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.063+0000 7f7da97fa640 10 client.?.objecter ms_handle_connect 0x7f7dac14be90 2024-03-25T18:13:52.064 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.063+0000 7f7da97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:52.066 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.065+0000 7f7da97fa640 10 client.?.objecter ms_handle_connect 0x7f7dac0b1730 2024-03-25T18:13:52.066 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.065+0000 7f7da97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:52.070 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.069+0000 7f7da97fa640 1 -- 172.21.15.66:0/3414616585 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 522666672 0 0) 0x7f7dac141ce0 con 0x7f7dac14be90 2024-03-25T18:13:52.070 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.069+0000 7f7da97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:13:52.070 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.069+0000 7f7da97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:52.070 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.069+0000 7f7da97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:52.070 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.069+0000 7f7da97fa640 10 monclient(hunting): my global_id is 4677 2024-03-25T18:13:52.070 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.069+0000 7f7da97fa640 1 -- 172.21.15.66:0/3414616585 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d8c004eb0 con 0x7f7dac14be90 2024-03-25T18:13:52.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.077+0000 7f7da97fa640 1 -- 172.21.15.66:0/3414616585 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3448256605 0 0) 0x7f7dac142610 con 0x7f7dac0b1730 2024-03-25T18:13:52.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.078+0000 7f7da97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:13:52.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.078+0000 7f7da97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:52.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.078+0000 7f7da97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:52.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.078+0000 7f7da97fa640 10 monclient(hunting): my global_id is 4672 2024-03-25T18:13:52.078 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.078+0000 7f7da97fa640 1 -- 172.21.15.66:0/3414616585 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dac141ce0 con 0x7f7dac0b1730 2024-03-25T18:13:52.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.088+0000 7f7da97fa640 1 -- 172.21.15.66:0/3414616585 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1962884470 0 0) 0x7f7d8c004eb0 con 0x7f7dac14be90 2024-03-25T18:13:52.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.088+0000 7f7da97fa640 1 -- 172.21.15.66:0/3414616585 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7dac142610 con 0x7f7dac14be90 2024-03-25T18:13:52.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.088+0000 7f7da97fa640 1 -- 172.21.15.66:0/3414616585 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7da4001910 con 0x7f7dac14be90 2024-03-25T18:13:52.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.088+0000 7f7da97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:52.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.088+0000 7f7da97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:52.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.088+0000 7f7da97fa640 10 monclient(hunting): dump: 2024-03-25T18:13:52.089 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:52.089 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:52.089 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:52.089 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:52.089 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:52.090 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:52.090 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:52.090 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:52.090 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:52.090 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:52.090 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.088+0000 7f7da97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:52.091 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.090+0000 7f7da97fa640 10 client.?.objecter ms_handle_connect 0x7f7dac0b4d60 2024-03-25T18:13:52.091 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.090+0000 7f7da97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:52.093 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.092+0000 7f7da97fa640 1 -- 172.21.15.66:0/3414616585 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1123336156 0 0) 0x7f7dac141ce0 con 0x7f7dac0b1730 2024-03-25T18:13:52.093 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.092+0000 7f7da97fa640 1 -- 172.21.15.66:0/3414616585 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d8c004eb0 con 0x7f7dac0b1730 2024-03-25T18:13:52.093 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.092+0000 7f7da97fa640 1 -- 172.21.15.66:0/3414616585 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7d94001b00 con 0x7f7dac0b1730 2024-03-25T18:13:52.093 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.092+0000 7f7da97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:52.093 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.092+0000 7f7da97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:52.093 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.092+0000 7f7da97fa640 10 monclient(hunting): dump: 2024-03-25T18:13:52.093 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:52.094 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:52.094 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:52.094 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:52.094 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:52.094 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:52.094 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:52.094 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:52.094 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:52.094 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:52.102 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.101+0000 7f7da97fa640 1 -- 172.21.15.66:0/3414616585 <== mon.0 v1:172.21.15.66:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3089131660 0 0) 0x7f7dac142610 con 0x7f7dac14be90 2024-03-25T18:13:52.102 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.101+0000 7f7da97fa640 1 -- 172.21.15.66:0/3414616585 >> v1:172.21.15.66:6790/0 conn(0x7f7dac0b4d60 legacy=0x7f7dac1db430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:52.102 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.101+0000 7f7da97fa640 1 -- 172.21.15.66:0/3414616585 >> v1:172.21.15.193:6789/0 conn(0x7f7dac0b1730 legacy=0x7f7dac142b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:52.102 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.101+0000 7f7da97fa640 10 monclient: _finish_hunting 0 2024-03-25T18:13:52.102 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.101+0000 7f7da97fa640 1 monclient: found mon.a 2024-03-25T18:13:52.103 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.101+0000 7f7da97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:52.103 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.101+0000 7f7da97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:13:52.103 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.101+0000 7f7da97fa640 1 -- 172.21.15.66:0/3414616585 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dac1db010 con 0x7f7dac14be90 2024-03-25T18:13:52.103 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.101+0000 7f7da97fa640 10 monclient: _finish_auth 0 2024-03-25T18:13:52.103 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.101+0000 7f7da97fa640 10 monclient: _check_auth_tickets 2024-03-25T18:13:52.103 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.101+0000 7f7da97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:52.103 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.101+0000 7f7db2f4b640 5 monclient: authenticate success, global_id 4677 2024-03-25T18:13:52.103 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.101+0000 7f7db2f4b640 10 monclient: _renew_subs 2024-03-25T18:13:52.103 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.101+0000 7f7db2f4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:13:52.103 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.101+0000 7f7db2f4b640 1 -- 172.21.15.66:0/3414616585 --> v1:172.21.15.66:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dac143240 con 0x7f7dac14be90 2024-03-25T18:13:52.103 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.101+0000 7f7db2f4b640 10 client.4677.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:52.103 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.101+0000 7f7db2f4b640 10 monclient: _renew_subs 2024-03-25T18:13:52.103 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.101+0000 7f7db2f4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:13:52.103 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.101+0000 7f7db2f4b640 1 -- 172.21.15.66:0/3414616585 --> v1:172.21.15.66:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dac13bd20 con 0x7f7dac14be90 2024-03-25T18:13:52.104 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.101+0000 7f7db2f4b640 1 librados: init done 2024-03-25T18:13:52.104 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.102+0000 7f7db2f4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:52.104 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.102+0000 7f7db2f4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:52.105 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.102+0000 7f7db2f4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:13:52.105 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.102+0000 7f7db2f4b640 1 -- 172.21.15.66:0/3414616585 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dac0627b0 con 0x7f7dac14be90 2024-03-25T18:13:52.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.451+0000 7f7da97fa640 1 -- 172.21.15.66:0/3414616585 <== mon.0 v1:172.21.15.66:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7da40035f0 con 0x7f7dac14be90 2024-03-25T18:13:52.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.451+0000 7f7da97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:52.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.451+0000 7f7da97fa640 1 -- 172.21.15.66:0/3414616585 <== mon.0 v1:172.21.15.66:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f7da4003a00 con 0x7f7dac14be90 2024-03-25T18:13:52.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.451+0000 7f7da97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:52.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.451+0000 7f7da97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:52.453 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.451+0000 7f7da97fa640 10 monclient: dump: 2024-03-25T18:13:52.453 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:52.453 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:52.453 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:52.453 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:52.453 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:52.453 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:52.453 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:52.453 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:52.453 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:52.453 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:52.453 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.451+0000 7f7daaffd640 4 set_mon_vals no callback set 2024-03-25T18:13:52.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.452+0000 7f7da97fa640 1 -- 172.21.15.66:0/3414616585 <== mon.0 v1:172.21.15.66:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f7da4003c20 con 0x7f7dac14be90 2024-03-25T18:13:52.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.453+0000 7f7db0ce7640 10 monclient: get_auth_request con 0x7f7d8c06de40 auth_method 0 2024-03-25T18:13:52.454 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.453+0000 7f7da97fa640 10 client.4677.objecter ms_handle_connect 0x7f7d8c06de40 2024-03-25T18:13:52.460 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.459+0000 7f7da97fa640 1 -- 172.21.15.66:0/3414616585 <== mon.0 v1:172.21.15.66:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4152745027 0 0) 0x7f7da4054660 con 0x7f7dac14be90 2024-03-25T18:13:52.460 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.459+0000 7f7da97fa640 10 client.4677.objecter ms_dispatch 0x7f7dac000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:13:52.460 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.459+0000 7f7da97fa640 3 client.4677.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:13:52.460 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.459+0000 7f7da97fa640 3 client.4677.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:13:52.460 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.459+0000 7f7da97fa640 20 client.4677.objecter dump_active .. 0 homeless 2024-03-25T18:13:52.460 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.459+0000 7f7da97fa640 1 -- 172.21.15.66:0/3414616585 <== mon.0 v1:172.21.15.66:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7da4050170 con 0x7f7dac14be90 2024-03-25T18:13:52.460 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.459+0000 7f7da97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:52.460 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.459+0000 7f7da97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:52.729 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.729+0000 7f7db2f4b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:52.729 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.729+0000 7f7db2f4b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:52.729 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.729+0000 7f7db2f4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:13:52.730 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.729+0000 7f7db2f4b640 1 -- 172.21.15.66:0/3414616585 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7dac137cc0 con 0x7f7dac14be90 2024-03-25T18:13:52.759 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.758+0000 7f7da97fa640 1 -- 172.21.15.66:0/3414616585 <== mon.0 v1:172.21.15.66:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4258808387) 0x7f7da40520a0 con 0x7f7dac14be90 2024-03-25T18:13:52.759 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.758+0000 7f7da97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:52.759 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.758+0000 7f7da97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:52.759 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:13:52.759 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":11,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:13:50.353085+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:50.353085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:52.761 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.760+0000 7f7db2f4b640 10 librados: watch_flush enter 2024-03-25T18:13:52.761 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.760+0000 7f7db2f4b640 10 librados: watch_flush exit 2024-03-25T18:13:52.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.760+0000 7f7db2f4b640 20 client.4677.objecter shutdown clearing up homeless session... 2024-03-25T18:13:52.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.760+0000 7f7db2f4b640 10 client.4677.objecter successfully canceled tick 2024-03-25T18:13:52.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.760+0000 7f7db2f4b640 1 -- 172.21.15.66:0/3414616585 >> v1:172.21.15.193:6800/2275676803 conn(0x7f7d8c06de40 legacy=0x7f7d8c070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:52.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.760+0000 7f7db2f4b640 10 monclient: shutdown 2024-03-25T18:13:52.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.760+0000 7f7db2f4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:52.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.760+0000 7f7db2f4b640 1 -- 172.21.15.66:0/3414616585 >> v1:172.21.15.66:6789/0 conn(0x7f7dac14be90 legacy=0x7f7dac140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:52.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.760+0000 7f7db2f4b640 1 -- 172.21.15.66:0/3414616585 shutdown_connections 2024-03-25T18:13:52.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.760+0000 7f7db2f4b640 1 -- 172.21.15.66:0/3414616585 >> 172.21.15.66:0/3414616585 conn(0x7f7dac0617a0 msgr2=0x7f7dac0b2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:52.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.760+0000 7f7db2f4b640 1 -- 172.21.15.66:0/3414616585 shutdown_connections 2024-03-25T18:13:52.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.760+0000 7f7db2f4b640 1 -- 172.21.15.66:0/3414616585 wait complete. 2024-03-25T18:13:52.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:52.761+0000 7f7db2f4b640 1 librados: shutdown 2024-03-25T18:13:52.772 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:59.577 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:59.687 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.686+0000 7fc4c579e640 10 monclient: get_monmap_and_config 2024-03-25T18:13:59.687 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.686+0000 7fc4c579e640 10 monclient: build_initial_monmap 2024-03-25T18:13:59.687 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.686+0000 7fc4c579e640 10 monclient: monmap: 2024-03-25T18:13:59.687 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:13:59.687 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:59.687 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:13:59.687313+0000 2024-03-25T18:13:59.687 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:13:59.687313+0000 2024-03-25T18:13:59.687 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:59.687 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:59.688 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:13:59.688 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:13:59.688 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:13:59.688 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:59.689 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.687+0000 7fc4c579e640 1 Processor -- start 2024-03-25T18:13:59.689 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.687+0000 7fc4c579e640 1 -- start start 2024-03-25T18:13:59.689 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.687+0000 7fc4c579e640 10 monclient: init 2024-03-25T18:13:59.690 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.688+0000 7fc4c579e640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:59.690 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.688+0000 7fc4c579e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:13:59.690 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.688+0000 7fc4c579e640 10 monclient(hunting): picked mon.noname-c con 0x7fc4c00b48e0 addr v1:172.21.15.193:6789/0 2024-03-25T18:13:59.690 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.688+0000 7fc4c579e640 10 monclient(hunting): picked mon.noname-b con 0x7fc4c013fc60 addr v1:172.21.15.66:6790/0 2024-03-25T18:13:59.690 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.688+0000 7fc4c579e640 10 monclient(hunting): picked mon.noname-a con 0x7fc4c00b3f90 addr v1:172.21.15.66:6789/0 2024-03-25T18:13:59.690 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.688+0000 7fc4c579e640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c00a9d10 con 0x7fc4c00b3f90 2024-03-25T18:13:59.690 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.688+0000 7fc4c579e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c00ab000 con 0x7fc4c00b48e0 2024-03-25T18:13:59.690 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.688+0000 7fc4c579e640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c00a9680 con 0x7fc4c013fc60 2024-03-25T18:13:59.690 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.688+0000 7fc4c579e640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:59.690 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.688+0000 7fc4c579e640 10 monclient(hunting): authenticate will time out at 7588.747492s 2024-03-25T18:13:59.690 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.688+0000 7fc4bf7fe640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7fc4c013fc60 0x7fc4c0145770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:60024/0 (socket says 172.21.15.66:60024) 2024-03-25T18:13:59.690 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.689+0000 7fc4bf7fe640 1 -- 172.21.15.66:0/3267312415 learned_addr learned my addr 172.21.15.66:0/3267312415 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:13:59.755 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.755+0000 7fc4beffd640 1 -- 172.21.15.66:0/3267312415 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 635378630 0 0) 0x7fc4c00a9d10 con 0x7fc4c00b3f90 2024-03-25T18:13:59.755 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.755+0000 7fc4beffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:59.755 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.755+0000 7fc4beffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:59.755 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.755+0000 7fc4beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:59.755 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.755+0000 7fc4beffd640 10 monclient(hunting): my global_id is 4680 2024-03-25T18:13:59.756 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.755+0000 7fc4beffd640 1 -- 172.21.15.66:0/3267312415 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4a8003520 con 0x7fc4c00b3f90 2024-03-25T18:13:59.769 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.768+0000 7fc4beffd640 1 -- 172.21.15.66:0/3267312415 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3718666715 0 0) 0x7fc4c00ab000 con 0x7fc4c00b48e0 2024-03-25T18:13:59.769 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.768+0000 7fc4beffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:59.769 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.768+0000 7fc4beffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:59.769 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.768+0000 7fc4beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:59.769 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.768+0000 7fc4beffd640 10 monclient(hunting): my global_id is 4675 2024-03-25T18:13:59.769 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.768+0000 7fc4beffd640 1 -- 172.21.15.66:0/3267312415 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4c00a9d10 con 0x7fc4c00b48e0 2024-03-25T18:13:59.774 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.773+0000 7fc4beffd640 1 -- 172.21.15.66:0/3267312415 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1154786589 0 0) 0x7fc4a8003520 con 0x7fc4c00b3f90 2024-03-25T18:13:59.774 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.773+0000 7fc4beffd640 1 -- 172.21.15.66:0/3267312415 >> v1:172.21.15.66:6790/0 conn(0x7fc4c013fc60 legacy=0x7fc4c0145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:59.774 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.773+0000 7fc4beffd640 1 -- 172.21.15.66:0/3267312415 >> v1:172.21.15.193:6789/0 conn(0x7fc4c00b48e0 legacy=0x7fc4c013f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:59.774 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.773+0000 7fc4beffd640 10 monclient: _finish_hunting 0 2024-03-25T18:13:59.774 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.773+0000 7fc4beffd640 1 monclient: found mon.noname-a 2024-03-25T18:13:59.774 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.773+0000 7fc4beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:59.774 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.773+0000 7fc4beffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.66:6789/0 2024-03-25T18:13:59.774 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.773+0000 7fc4beffd640 1 -- 172.21.15.66:0/3267312415 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c00a9810 con 0x7fc4c00b3f90 2024-03-25T18:13:59.774 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.773+0000 7fc4beffd640 10 monclient: _finish_auth 0 2024-03-25T18:13:59.774 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.773+0000 7fc4beffd640 10 monclient: _check_auth_tickets 2024-03-25T18:13:59.774 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.773+0000 7fc4beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:59.774 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.773+0000 7fc4c579e640 5 monclient: authenticate success, global_id 4680 2024-03-25T18:13:59.775 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.773+0000 7fc4beffd640 1 -- 172.21.15.66:0/3267312415 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fc4b8001a10 con 0x7fc4c00b3f90 2024-03-25T18:13:59.775 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.773+0000 7fc4beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:59.775 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.773+0000 7fc4beffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:59.775 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.773+0000 7fc4beffd640 10 monclient: dump: 2024-03-25T18:13:59.775 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:59.775 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:59.775 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:59.775 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:59.775 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:59.775 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:59.775 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:59.775 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:59.775 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:59.775 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:59.775 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.773+0000 7fc4c579e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:59.786 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.786+0000 7fc4beffd640 1 -- 172.21.15.66:0/3267312415 <== mon.0 v1:172.21.15.66:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4b8001b70 con 0x7fc4c00b3f90 2024-03-25T18:13:59.787 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.786+0000 7fc4beffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:59.787 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.786+0000 7fc4beffd640 1 -- 172.21.15.66:0/3267312415 <== mon.0 v1:172.21.15.66:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fc4b8001f80 con 0x7fc4c00b3f90 2024-03-25T18:13:59.787 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.786+0000 7fc4c579e640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:59.787 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.786+0000 7fc4c579e640 4 set_mon_vals no callback set 2024-03-25T18:13:59.787 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.786+0000 7fc4beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:59.787 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.786+0000 7fc4beffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:59.787 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.786+0000 7fc4beffd640 10 monclient: dump: 2024-03-25T18:13:59.787 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:59.787 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:59.787 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:59.787 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:59.787 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:59.787 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:59.787 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:59.787 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:59.788 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:59.788 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:59.789 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.787+0000 7fc4c579e640 10 monclient: shutdown 2024-03-25T18:13:59.789 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.787+0000 7fc4c579e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:59.789 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.787+0000 7fc4c579e640 1 -- 172.21.15.66:0/3267312415 >> v1:172.21.15.66:6789/0 conn(0x7fc4c00b3f90 legacy=0x7fc4c014dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:59.789 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.787+0000 7fc4c579e640 1 -- 172.21.15.66:0/3267312415 shutdown_connections 2024-03-25T18:13:59.789 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.787+0000 7fc4c579e640 1 -- 172.21.15.66:0/3267312415 >> 172.21.15.66:0/3267312415 conn(0x7fc4c00615c0 msgr2=0x7fc4c00619d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:59.789 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.787+0000 7fc4c579e640 1 -- 172.21.15.66:0/3267312415 shutdown_connections 2024-03-25T18:13:59.789 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.787+0000 7fc4c579e640 1 -- 172.21.15.66:0/3267312415 wait complete. 2024-03-25T18:13:59.789 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.788+0000 7fc4c579e640 10 monclient: build_initial_monmap 2024-03-25T18:13:59.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.788+0000 7fc4c579e640 10 monclient: monmap: 2024-03-25T18:13:59.790 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:13:59.790 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:59.790 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:13:59.790 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:13:59.790 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:59.790 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:13:59.790 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:13:59.790 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:13:59.790 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:13:59.790 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:59.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.788+0000 7fc4c579e640 1 librados: starting msgr at 2024-03-25T18:13:59.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.788+0000 7fc4c579e640 1 librados: starting objecter 2024-03-25T18:13:59.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.789+0000 7fc4c579e640 1 Processor -- start 2024-03-25T18:13:59.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.789+0000 7fc4c579e640 1 -- start start 2024-03-25T18:13:59.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.789+0000 7fc4c579e640 1 librados: setting wanted keys 2024-03-25T18:13:59.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.789+0000 7fc4c579e640 1 librados: calling monclient init 2024-03-25T18:13:59.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.789+0000 7fc4c579e640 10 monclient: init 2024-03-25T18:13:59.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.790+0000 7fc4c579e640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:59.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.790+0000 7fc4c579e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:59.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.790+0000 7fc4c579e640 10 monclient(hunting): picked mon.noname-c con 0x7fc4c00b3f90 addr v1:172.21.15.66:6790/0 2024-03-25T18:13:59.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.790+0000 7fc4c579e640 10 monclient(hunting): picked mon.noname-b con 0x7fc4c00b48e0 addr v1:172.21.15.193:6789/0 2024-03-25T18:13:59.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.790+0000 7fc4c579e640 10 monclient(hunting): picked mon.noname-a con 0x7fc4c013fc60 addr v1:172.21.15.66:6789/0 2024-03-25T18:13:59.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.790+0000 7fc4c579e640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c01ceaa0 con 0x7fc4c013fc60 2024-03-25T18:13:59.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.790+0000 7fc4c579e640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c00abaa0 con 0x7fc4c00b48e0 2024-03-25T18:13:59.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.790+0000 7fc4c579e640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c00abd80 con 0x7fc4c00b3f90 2024-03-25T18:13:59.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.790+0000 7fc4c579e640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:59.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.790+0000 7fc4c579e640 10 monclient(hunting): authenticate will time out at 7588.849513s 2024-03-25T18:13:59.806 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.806+0000 7fc4c4f9d640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7fc4c013fc60 0x7fc4c01cb240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:55954/0 (socket says 172.21.15.66:55954) 2024-03-25T18:13:59.807 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.806+0000 7fc4c4f9d640 1 -- 172.21.15.66:0/3203731850 learned_addr learned my addr 172.21.15.66:0/3203731850 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:13:59.809 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.808+0000 7fc4bcff9640 10 client.?.objecter ms_handle_connect 0x7fc4c00b48e0 2024-03-25T18:13:59.809 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.808+0000 7fc4bcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:59.809 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.809+0000 7fc4bcff9640 10 client.?.objecter ms_handle_connect 0x7fc4c013fc60 2024-03-25T18:13:59.809 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.809+0000 7fc4bcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:59.820 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.820+0000 7fc4bcff9640 10 client.?.objecter ms_handle_connect 0x7fc4c00b3f90 2024-03-25T18:13:59.820 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.820+0000 7fc4bcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:59.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.827+0000 7fc4bcff9640 1 -- 172.21.15.66:0/3203731850 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3535774738 0 0) 0x7fc4c00abaa0 con 0x7fc4c00b48e0 2024-03-25T18:13:59.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.827+0000 7fc4bcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:13:59.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.827+0000 7fc4bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:59.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.827+0000 7fc4bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:59.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.827+0000 7fc4bcff9640 10 monclient(hunting): my global_id is 4678 2024-03-25T18:13:59.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.827+0000 7fc4bcff9640 1 -- 172.21.15.66:0/3203731850 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4ac004eb0 con 0x7fc4c00b48e0 2024-03-25T18:13:59.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.829+0000 7fc4bcff9640 1 -- 172.21.15.66:0/3203731850 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3331573169 0 0) 0x7fc4c01ceaa0 con 0x7fc4c013fc60 2024-03-25T18:13:59.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.829+0000 7fc4bcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:13:59.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.829+0000 7fc4bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:59.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.829+0000 7fc4bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:59.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.829+0000 7fc4bcff9640 10 monclient(hunting): my global_id is 4683 2024-03-25T18:13:59.830 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.829+0000 7fc4bcff9640 1 -- 172.21.15.66:0/3203731850 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4c00abaa0 con 0x7fc4c013fc60 2024-03-25T18:13:59.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.844+0000 7fc4bcff9640 1 -- 172.21.15.66:0/3203731850 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3234428842 0 0) 0x7fc4c00abaa0 con 0x7fc4c013fc60 2024-03-25T18:13:59.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.845+0000 7fc4bcff9640 1 -- 172.21.15.66:0/3203731850 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4c01ceaa0 con 0x7fc4c013fc60 2024-03-25T18:13:59.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.845+0000 7fc4bcff9640 1 -- 172.21.15.66:0/3203731850 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fc4b8001ec0 con 0x7fc4c013fc60 2024-03-25T18:13:59.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.845+0000 7fc4bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:59.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.845+0000 7fc4bcff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:59.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.845+0000 7fc4bcff9640 10 monclient(hunting): dump: 2024-03-25T18:13:59.846 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:59.846 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:59.846 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:59.846 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:59.846 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:59.846 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:59.846 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:59.846 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:59.846 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:59.846 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:59.846 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.845+0000 7fc4bcff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:59.847 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.845+0000 7fc4bcff9640 1 -- 172.21.15.66:0/3203731850 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1135301962 0 0) 0x7fc4ac004eb0 con 0x7fc4c00b48e0 2024-03-25T18:13:59.847 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.846+0000 7fc4bcff9640 1 -- 172.21.15.66:0/3203731850 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4c00abaa0 con 0x7fc4c00b48e0 2024-03-25T18:13:59.847 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.846+0000 7fc4bcff9640 1 -- 172.21.15.66:0/3203731850 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fc4b0001800 con 0x7fc4c00b48e0 2024-03-25T18:13:59.847 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.846+0000 7fc4bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:59.847 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.846+0000 7fc4bcff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:59.847 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.846+0000 7fc4bcff9640 10 monclient(hunting): dump: 2024-03-25T18:13:59.847 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:13:59.847 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:13:59.847 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:59.848 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:13:59.848 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:59.848 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:13:59.848 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:13:59.848 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:13:59.848 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:13:59.848 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:13:59.849 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.849+0000 7fc4bcff9640 1 -- 172.21.15.66:0/3203731850 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 684469450 0 0) 0x7fc4c00abd80 con 0x7fc4c00b3f90 2024-03-25T18:13:59.849 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.849+0000 7fc4bcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:13:59.850 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.849+0000 7fc4bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:59.850 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.849+0000 7fc4bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:59.850 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.849+0000 7fc4bcff9640 10 monclient(hunting): my global_id is 4676 2024-03-25T18:13:59.850 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.849+0000 7fc4bcff9640 1 -- 172.21.15.66:0/3203731850 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4ac004eb0 con 0x7fc4c00b3f90 2024-03-25T18:13:59.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.855+0000 7fc4bcff9640 1 -- 172.21.15.66:0/3203731850 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 197063704 0 0) 0x7fc4c00abaa0 con 0x7fc4c00b48e0 2024-03-25T18:13:59.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.855+0000 7fc4bcff9640 1 -- 172.21.15.66:0/3203731850 >> v1:172.21.15.66:6790/0 conn(0x7fc4c00b3f90 legacy=0x7fc4c01cdd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:59.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.855+0000 7fc4bcff9640 1 -- 172.21.15.66:0/3203731850 >> v1:172.21.15.66:6789/0 conn(0x7fc4c013fc60 legacy=0x7fc4c01cb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:59.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.855+0000 7fc4bcff9640 10 monclient: _finish_hunting 0 2024-03-25T18:13:59.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.855+0000 7fc4bcff9640 1 monclient: found mon.b 2024-03-25T18:13:59.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.855+0000 7fc4bcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:59.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.855+0000 7fc4bcff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:13:59.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.855+0000 7fc4bcff9640 1 -- 172.21.15.66:0/3203731850 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c00ac060 con 0x7fc4c00b48e0 2024-03-25T18:13:59.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.855+0000 7fc4bcff9640 10 monclient: _finish_auth 0 2024-03-25T18:13:59.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.855+0000 7fc4bcff9640 10 monclient: _check_auth_tickets 2024-03-25T18:13:59.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.855+0000 7fc4bcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:59.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.855+0000 7fc4c579e640 5 monclient: authenticate success, global_id 4678 2024-03-25T18:13:59.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.855+0000 7fc4c579e640 10 monclient: _renew_subs 2024-03-25T18:13:59.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.855+0000 7fc4c579e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:13:59.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.855+0000 7fc4c579e640 1 -- 172.21.15.66:0/3203731850 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4c01ce420 con 0x7fc4c00b48e0 2024-03-25T18:13:59.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.856+0000 7fc4c579e640 10 client.4678.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:59.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.856+0000 7fc4c579e640 10 monclient: _renew_subs 2024-03-25T18:13:59.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.856+0000 7fc4c579e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:13:59.857 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.856+0000 7fc4c579e640 1 -- 172.21.15.66:0/3203731850 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4c013b0c0 con 0x7fc4c00b48e0 2024-03-25T18:13:59.858 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.856+0000 7fc4c579e640 1 librados: init done 2024-03-25T18:13:59.858 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.857+0000 7fc4c579e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:59.858 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.857+0000 7fc4c579e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:59.858 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.857+0000 7fc4c579e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:13:59.858 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:13:59.857+0000 7fc4c579e640 1 -- 172.21.15.66:0/3203731850 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4c00627b0 con 0x7fc4c00b48e0 2024-03-25T18:14:00.143 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.142+0000 7fc4bcff9640 1 -- 172.21.15.66:0/3203731850 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4b0001d10 con 0x7fc4c00b48e0 2024-03-25T18:14:00.143 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.142+0000 7fc4bcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:00.143 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.142+0000 7fc4bcff9640 1 -- 172.21.15.66:0/3203731850 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fc4b0004980 con 0x7fc4c00b48e0 2024-03-25T18:14:00.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.142+0000 7fc4bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:00.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.142+0000 7fc4bcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:00.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.142+0000 7fc4bcff9640 10 monclient: dump: 2024-03-25T18:14:00.144 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:00.144 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:00.144 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:00.144 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:00.144 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:00.144 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:00.144 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:00.144 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:00.144 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:00.144 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:00.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.142+0000 7fc4be7fc640 4 set_mon_vals no callback set 2024-03-25T18:14:00.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.142+0000 7fc4bcff9640 1 -- 172.21.15.66:0/3203731850 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7fc4b001a950 con 0x7fc4c00b48e0 2024-03-25T18:14:00.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.143+0000 7fc4bffff640 10 monclient: get_auth_request con 0x7fc4ac06df00 auth_method 0 2024-03-25T18:14:00.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.143+0000 7fc4bcff9640 10 client.4678.objecter ms_handle_connect 0x7fc4ac06df00 2024-03-25T18:14:00.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.148+0000 7fc4bcff9640 1 -- 172.21.15.66:0/3203731850 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4152745027 0 0) 0x7fc4b00546d0 con 0x7fc4c00b48e0 2024-03-25T18:14:00.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.148+0000 7fc4bcff9640 10 client.4678.objecter ms_dispatch 0x7fc4c0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:14:00.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.148+0000 7fc4bcff9640 3 client.4678.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:14:00.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.148+0000 7fc4bcff9640 3 client.4678.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:14:00.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.148+0000 7fc4bcff9640 20 client.4678.objecter dump_active .. 0 homeless 2024-03-25T18:14:00.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.149+0000 7fc4bcff9640 1 -- 172.21.15.66:0/3203731850 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc4b0050010 con 0x7fc4c00b48e0 2024-03-25T18:14:00.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.149+0000 7fc4bcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:00.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.149+0000 7fc4bcff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:00.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.412+0000 7fc4c579e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:00.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.412+0000 7fc4c579e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:00.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.412+0000 7fc4c579e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:14:00.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.412+0000 7fc4c579e640 1 -- 172.21.15.66:0/3203731850 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4c01cec50 con 0x7fc4c00b48e0 2024-03-25T18:14:00.433 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.432+0000 7fc4bcff9640 1 -- 172.21.15.66:0/3203731850 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4258808387) 0x7fc4c01cec50 con 0x7fc4c00b48e0 2024-03-25T18:14:00.433 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.432+0000 7fc4bcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:00.433 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.432+0000 7fc4bcff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:00.434 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:14:00.434 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":11,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:13:50.353085+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:50.353085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:00.436 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.435+0000 7fc4c579e640 10 librados: watch_flush enter 2024-03-25T18:14:00.436 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.435+0000 7fc4c579e640 10 librados: watch_flush exit 2024-03-25T18:14:00.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.435+0000 7fc4c579e640 20 client.4678.objecter shutdown clearing up homeless session... 2024-03-25T18:14:00.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.435+0000 7fc4c579e640 10 client.4678.objecter successfully canceled tick 2024-03-25T18:14:00.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.435+0000 7fc4c579e640 1 -- 172.21.15.66:0/3203731850 >> v1:172.21.15.193:6800/2275676803 conn(0x7fc4ac06df00 legacy=0x7fc4ac0703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:00.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.435+0000 7fc4c579e640 10 monclient: shutdown 2024-03-25T18:14:00.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.435+0000 7fc4c579e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:00.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.435+0000 7fc4c579e640 1 -- 172.21.15.66:0/3203731850 >> v1:172.21.15.193:6789/0 conn(0x7fc4c00b48e0 legacy=0x7fc4c013a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:00.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.435+0000 7fc4c579e640 1 -- 172.21.15.66:0/3203731850 shutdown_connections 2024-03-25T18:14:00.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.435+0000 7fc4c579e640 1 -- 172.21.15.66:0/3203731850 >> 172.21.15.66:0/3203731850 conn(0x7fc4c00615c0 msgr2=0x7fc4c00b1e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:00.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.436+0000 7fc4c579e640 1 -- 172.21.15.66:0/3203731850 shutdown_connections 2024-03-25T18:14:00.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.436+0000 7fc4c579e640 1 -- 172.21.15.66:0/3203731850 wait complete. 2024-03-25T18:14:00.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:00.436+0000 7fc4c579e640 1 librados: shutdown 2024-03-25T18:14:00.449 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:07.255 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:07.374 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.373+0000 7f2468579640 10 monclient: get_monmap_and_config 2024-03-25T18:14:07.374 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.373+0000 7f2468579640 10 monclient: build_initial_monmap 2024-03-25T18:14:07.375 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.373+0000 7f2468579640 10 monclient: monmap: 2024-03-25T18:14:07.375 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:14:07.375 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:07.375 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:14:07.374373+0000 2024-03-25T18:14:07.375 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:14:07.374373+0000 2024-03-25T18:14:07.375 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:07.375 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:07.375 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:14:07.375 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:14:07.375 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:14:07.375 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:07.376 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.374+0000 7f2468579640 1 Processor -- start 2024-03-25T18:14:07.376 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.374+0000 7f2468579640 1 -- start start 2024-03-25T18:14:07.376 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.374+0000 7f2468579640 10 monclient: init 2024-03-25T18:14:07.376 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.375+0000 7f2468579640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:07.377 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.375+0000 7f2468579640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:07.377 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.375+0000 7f2468579640 10 monclient(hunting): picked mon.noname-a con 0x7f24600a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:14:07.377 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.375+0000 7f2468579640 10 monclient(hunting): picked mon.noname-c con 0x7f246013f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:14:07.377 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.375+0000 7f2468579640 10 monclient(hunting): picked mon.noname-b con 0x7f24601430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:14:07.377 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.375+0000 7f2468579640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2460142520 con 0x7f24600a98f0 2024-03-25T18:14:07.377 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.375+0000 7f2468579640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24600b28a0 con 0x7f246013f710 2024-03-25T18:14:07.377 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.375+0000 7f2468579640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24600b2a30 con 0x7f24601430c0 2024-03-25T18:14:07.377 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.375+0000 7f2468579640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:07.377 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.375+0000 7f2468579640 10 monclient(hunting): authenticate will time out at 7596.434957s 2024-03-25T18:14:07.385 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.385+0000 7f2466315640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f24600a98f0 0x7f24600a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:52058/0 (socket says 172.21.15.66:52058) 2024-03-25T18:14:07.385 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.385+0000 7f2466315640 1 -- 172.21.15.66:0/3293783365 learned_addr learned my addr 172.21.15.66:0/3293783365 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:14:07.391 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.391+0000 7f2465313640 1 -- 172.21.15.66:0/3293783365 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3400772301 0 0) 0x7f2460142520 con 0x7f24600a98f0 2024-03-25T18:14:07.392 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.391+0000 7f2465313640 10 monclient(hunting): _negotiate 2024-03-25T18:14:07.392 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.391+0000 7f2465313640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:07.392 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.391+0000 7f2465313640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:07.392 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.391+0000 7f2465313640 10 monclient(hunting): my global_id is 4686 2024-03-25T18:14:07.392 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.391+0000 7f2465313640 1 -- 172.21.15.66:0/3293783365 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f244c003520 con 0x7f24600a98f0 2024-03-25T18:14:07.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.404+0000 7f2465313640 1 -- 172.21.15.66:0/3293783365 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 311228950 0 0) 0x7f244c003520 con 0x7f24600a98f0 2024-03-25T18:14:07.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.404+0000 7f2465313640 1 -- 172.21.15.66:0/3293783365 >> v1:172.21.15.66:6790/0 conn(0x7f24601430c0 legacy=0x7f246014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:07.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.404+0000 7f2465313640 1 -- 172.21.15.66:0/3293783365 >> v1:172.21.15.193:6789/0 conn(0x7f246013f710 legacy=0x7f24601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:07.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.404+0000 7f2465313640 10 monclient: _finish_hunting 0 2024-03-25T18:14:07.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.404+0000 7f2465313640 1 monclient: found mon.noname-a 2024-03-25T18:14:07.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.404+0000 7f2465313640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:07.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.404+0000 7f2465313640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.66:6789/0 2024-03-25T18:14:07.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.404+0000 7f2465313640 1 -- 172.21.15.66:0/3293783365 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24600b44c0 con 0x7f24600a98f0 2024-03-25T18:14:07.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.404+0000 7f2465313640 10 monclient: _finish_auth 0 2024-03-25T18:14:07.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.404+0000 7f2465313640 10 monclient: _check_auth_tickets 2024-03-25T18:14:07.405 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.404+0000 7f2465313640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:07.406 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.404+0000 7f2465313640 1 -- 172.21.15.66:0/3293783365 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f245c001690 con 0x7f24600a98f0 2024-03-25T18:14:07.406 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.404+0000 7f2468579640 5 monclient: authenticate success, global_id 4686 2024-03-25T18:14:07.406 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.404+0000 7f2468579640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:07.406 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.404+0000 7f2465313640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:07.406 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.404+0000 7f2465313640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:07.406 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.404+0000 7f2465313640 10 monclient: dump: 2024-03-25T18:14:07.406 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:07.406 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:07.406 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:07.406 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:07.406 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:07.406 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:07.406 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:07.406 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:07.407 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:07.407 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:07.407 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.404+0000 7f2468579640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:07.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.411+0000 7f2465313640 1 -- 172.21.15.66:0/3293783365 <== mon.0 v1:172.21.15.66:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f245c0017f0 con 0x7f24600a98f0 2024-03-25T18:14:07.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.411+0000 7f2465313640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:07.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.411+0000 7f2468579640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:07.411 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.411+0000 7f2468579640 4 set_mon_vals no callback set 2024-03-25T18:14:07.412 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.411+0000 7f2468579640 10 monclient: shutdown 2024-03-25T18:14:07.412 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.411+0000 7f2468579640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:07.412 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.411+0000 7f2468579640 1 -- 172.21.15.66:0/3293783365 >> v1:172.21.15.66:6789/0 conn(0x7f24600a98f0 legacy=0x7f24600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:07.412 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.412+0000 7f2468579640 1 -- 172.21.15.66:0/3293783365 shutdown_connections 2024-03-25T18:14:07.412 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.412+0000 7f2468579640 1 -- 172.21.15.66:0/3293783365 >> 172.21.15.66:0/3293783365 conn(0x7f24600b7ea0 msgr2=0x7f24600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:07.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.412+0000 7f2468579640 1 -- 172.21.15.66:0/3293783365 shutdown_connections 2024-03-25T18:14:07.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.412+0000 7f2468579640 1 -- 172.21.15.66:0/3293783365 wait complete. 2024-03-25T18:14:07.413 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.412+0000 7f2468579640 10 monclient: build_initial_monmap 2024-03-25T18:14:07.414 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.412+0000 7f2468579640 10 monclient: monmap: 2024-03-25T18:14:07.414 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:14:07.414 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:07.414 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:14:07.414 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:14:07.414 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:07.414 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:14:07.414 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:14:07.414 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:14:07.414 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:14:07.414 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:07.414 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.413+0000 7f2468579640 1 librados: starting msgr at 2024-03-25T18:14:07.414 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.413+0000 7f2468579640 1 librados: starting objecter 2024-03-25T18:14:07.415 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.413+0000 7f2468579640 1 Processor -- start 2024-03-25T18:14:07.415 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.413+0000 7f2468579640 1 -- start start 2024-03-25T18:14:07.415 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.413+0000 7f2468579640 1 librados: setting wanted keys 2024-03-25T18:14:07.415 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.413+0000 7f2468579640 1 librados: calling monclient init 2024-03-25T18:14:07.415 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.413+0000 7f2468579640 10 monclient: init 2024-03-25T18:14:07.416 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.414+0000 7f2468579640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:07.416 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.414+0000 7f2468579640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:07.416 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.414+0000 7f2468579640 10 monclient(hunting): picked mon.noname-b con 0x7f24600a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:14:07.416 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.414+0000 7f2468579640 10 monclient(hunting): picked mon.noname-a con 0x7f246013f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:14:07.416 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.414+0000 7f2468579640 10 monclient(hunting): picked mon.noname-c con 0x7f24601430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:14:07.416 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.414+0000 7f2468579640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24600abb00 con 0x7f246013f710 2024-03-25T18:14:07.416 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.414+0000 7f2468579640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24601d51f0 con 0x7f24600a98f0 2024-03-25T18:14:07.416 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.414+0000 7f2468579640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24601d7730 con 0x7f24601430c0 2024-03-25T18:14:07.416 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.414+0000 7f2468579640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:07.416 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.414+0000 7f2468579640 10 monclient(hunting): authenticate will time out at 7596.473924s 2024-03-25T18:14:07.423 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.422+0000 7f2465b14640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f246013f710 0x7f24601d7c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:52110/0 (socket says 172.21.15.66:52110) 2024-03-25T18:14:07.423 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.422+0000 7f2465b14640 1 -- 172.21.15.66:0/419645173 learned_addr learned my addr 172.21.15.66:0/419645173 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:14:07.425 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.424+0000 7f2456ffd640 10 client.?.objecter ms_handle_connect 0x7f246013f710 2024-03-25T18:14:07.425 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.424+0000 7f2456ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:07.436 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.435+0000 7f2456ffd640 1 -- 172.21.15.66:0/419645173 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3406533465 0 0) 0x7f24600abb00 con 0x7f246013f710 2024-03-25T18:14:07.436 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.435+0000 7f2456ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:07.436 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.435+0000 7f2456ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:07.436 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.435+0000 7f2456ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:07.436 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.435+0000 7f2456ffd640 10 monclient(hunting): my global_id is 4689 2024-03-25T18:14:07.436 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.435+0000 7f2456ffd640 1 -- 172.21.15.66:0/419645173 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2444004eb0 con 0x7f246013f710 2024-03-25T18:14:07.436 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.436+0000 7f2456ffd640 10 client.?.objecter ms_handle_connect 0x7f24601430c0 2024-03-25T18:14:07.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.436+0000 7f2456ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:07.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.437+0000 7f2456ffd640 10 client.?.objecter ms_handle_connect 0x7f24600a98f0 2024-03-25T18:14:07.438 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.437+0000 7f2456ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:07.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.449+0000 7f2456ffd640 1 -- 172.21.15.66:0/419645173 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 793377033 0 0) 0x7f24601d51f0 con 0x7f24600a98f0 2024-03-25T18:14:07.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.449+0000 7f2456ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:07.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.449+0000 7f2456ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:07.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.449+0000 7f2456ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:07.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.449+0000 7f2456ffd640 10 monclient(hunting): my global_id is 4684 2024-03-25T18:14:07.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.449+0000 7f2456ffd640 1 -- 172.21.15.66:0/419645173 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24600abb00 con 0x7f24600a98f0 2024-03-25T18:14:07.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.452+0000 7f2456ffd640 1 -- 172.21.15.66:0/419645173 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4095951603 0 0) 0x7f24601d7730 con 0x7f24601430c0 2024-03-25T18:14:07.453 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.452+0000 7f2456ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:07.453 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.452+0000 7f2456ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:07.453 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.452+0000 7f2456ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:07.453 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.452+0000 7f2456ffd640 10 monclient(hunting): my global_id is 4682 2024-03-25T18:14:07.453 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.452+0000 7f2456ffd640 1 -- 172.21.15.66:0/419645173 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24601d51f0 con 0x7f24601430c0 2024-03-25T18:14:07.464 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.464+0000 7f2456ffd640 1 -- 172.21.15.66:0/419645173 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3961995134 0 0) 0x7f2444004eb0 con 0x7f246013f710 2024-03-25T18:14:07.464 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.464+0000 7f2456ffd640 1 -- 172.21.15.66:0/419645173 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24601d7730 con 0x7f246013f710 2024-03-25T18:14:07.464 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.464+0000 7f2456ffd640 1 -- 172.21.15.66:0/419645173 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f2450001a30 con 0x7f246013f710 2024-03-25T18:14:07.464 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.464+0000 7f2456ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:07.464 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.464+0000 7f2456ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:07.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.464+0000 7f2456ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:07.465 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:07.465 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:07.465 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:07.465 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:07.465 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:07.465 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:07.465 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:07.465 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:07.465 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:07.465 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:07.465 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.464+0000 7f2456ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:07.468 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.468+0000 7f2456ffd640 1 -- 172.21.15.66:0/419645173 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1776396608 0 0) 0x7f24601d51f0 con 0x7f24601430c0 2024-03-25T18:14:07.468 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.468+0000 7f2456ffd640 1 -- 172.21.15.66:0/419645173 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2444004eb0 con 0x7f24601430c0 2024-03-25T18:14:07.468 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.468+0000 7f2456ffd640 1 -- 172.21.15.66:0/419645173 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f2458001a90 con 0x7f24601430c0 2024-03-25T18:14:07.468 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.468+0000 7f2456ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:07.468 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.468+0000 7f2456ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:07.469 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.468+0000 7f2456ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:07.469 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:07.469 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:07.469 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:07.469 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:07.469 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:07.469 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:07.469 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:07.469 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:07.469 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:07.469 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:07.470 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.470+0000 7f2456ffd640 1 -- 172.21.15.66:0/419645173 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 396409492 0 0) 0x7f24600abb00 con 0x7f24600a98f0 2024-03-25T18:14:07.470 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.470+0000 7f2456ffd640 1 -- 172.21.15.66:0/419645173 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24601d51f0 con 0x7f24600a98f0 2024-03-25T18:14:07.470 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.470+0000 7f2456ffd640 1 -- 172.21.15.66:0/419645173 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f245c001ca0 con 0x7f24600a98f0 2024-03-25T18:14:07.470 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.470+0000 7f2456ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:07.470 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.470+0000 7f2456ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:07.471 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.470+0000 7f2456ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:07.471 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:07.471 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:07.471 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:07.471 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:07.471 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:07.471 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:07.471 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:07.471 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:07.471 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:07.471 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:07.477 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.476+0000 7f2456ffd640 1 -- 172.21.15.66:0/419645173 <== mon.0 v1:172.21.15.66:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2946838699 0 0) 0x7f24601d7730 con 0x7f246013f710 2024-03-25T18:14:07.477 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.476+0000 7f2456ffd640 1 -- 172.21.15.66:0/419645173 >> v1:172.21.15.66:6790/0 conn(0x7f24601430c0 legacy=0x7f24601e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:07.477 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.476+0000 7f2456ffd640 1 -- 172.21.15.66:0/419645173 >> v1:172.21.15.193:6789/0 conn(0x7f24600a98f0 legacy=0x7f24601384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:07.477 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.476+0000 7f2456ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:07.477 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.476+0000 7f2456ffd640 1 monclient: found mon.a 2024-03-25T18:14:07.477 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.476+0000 7f2456ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:07.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.476+0000 7f2456ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:14:07.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.476+0000 7f2456ffd640 1 -- 172.21.15.66:0/419645173 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2460138be0 con 0x7f246013f710 2024-03-25T18:14:07.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.476+0000 7f2456ffd640 10 monclient: _finish_auth 0 2024-03-25T18:14:07.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.476+0000 7f2456ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:07.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.476+0000 7f2456ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:07.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.476+0000 7f2468579640 5 monclient: authenticate success, global_id 4689 2024-03-25T18:14:07.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.476+0000 7f2468579640 10 monclient: _renew_subs 2024-03-25T18:14:07.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.476+0000 7f2468579640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:14:07.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.476+0000 7f2468579640 1 -- 172.21.15.66:0/419645173 --> v1:172.21.15.66:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24601d8360 con 0x7f246013f710 2024-03-25T18:14:07.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.476+0000 7f2468579640 10 client.4689.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:07.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.476+0000 7f2468579640 10 monclient: _renew_subs 2024-03-25T18:14:07.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.476+0000 7f2468579640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:14:07.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.476+0000 7f2468579640 1 -- 172.21.15.66:0/419645173 --> v1:172.21.15.66:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2460137f40 con 0x7f246013f710 2024-03-25T18:14:07.478 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.476+0000 7f2468579640 1 librados: init done 2024-03-25T18:14:07.479 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.477+0000 7f2468579640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:07.479 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.477+0000 7f2468579640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:07.479 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.477+0000 7f2468579640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:14:07.479 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.477+0000 7f2468579640 1 -- 172.21.15.66:0/419645173 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24601d7590 con 0x7f246013f710 2024-03-25T18:14:07.787 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.787+0000 7f2456ffd640 1 -- 172.21.15.66:0/419645173 <== mon.0 v1:172.21.15.66:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2450001c70 con 0x7f246013f710 2024-03-25T18:14:07.787 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.787+0000 7f2456ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:07.788 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.787+0000 7f2456ffd640 1 -- 172.21.15.66:0/419645173 <== mon.0 v1:172.21.15.66:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f24500038c0 con 0x7f246013f710 2024-03-25T18:14:07.788 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.787+0000 7f2456ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:07.788 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.787+0000 7f2456ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:07.788 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.787+0000 7f2456ffd640 10 monclient: dump: 2024-03-25T18:14:07.788 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:07.788 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:07.788 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:07.788 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:07.788 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:07.788 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:07.788 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:07.788 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:07.788 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:07.788 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:07.788 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.787+0000 7f2464b12640 4 set_mon_vals no callback set 2024-03-25T18:14:07.789 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.788+0000 7f2456ffd640 1 -- 172.21.15.66:0/419645173 <== mon.0 v1:172.21.15.66:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f2450003aa0 con 0x7f246013f710 2024-03-25T18:14:07.789 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.789+0000 7f2466315640 10 monclient: get_auth_request con 0x7f244406e0f0 auth_method 0 2024-03-25T18:14:07.789 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.789+0000 7f2456ffd640 10 client.4689.objecter ms_handle_connect 0x7f244406e0f0 2024-03-25T18:14:07.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.793+0000 7f2456ffd640 1 -- 172.21.15.66:0/419645173 <== mon.0 v1:172.21.15.66:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4152745027 0 0) 0x7f24500546f0 con 0x7f246013f710 2024-03-25T18:14:07.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.794+0000 7f2456ffd640 10 client.4689.objecter ms_dispatch 0x7f2460000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:14:07.794 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.794+0000 7f2456ffd640 3 client.4689.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:14:07.795 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.794+0000 7f2456ffd640 3 client.4689.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:14:07.795 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.794+0000 7f2456ffd640 20 client.4689.objecter dump_active .. 0 homeless 2024-03-25T18:14:07.795 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.794+0000 7f2456ffd640 1 -- 172.21.15.66:0/419645173 <== mon.0 v1:172.21.15.66:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2450050200 con 0x7f246013f710 2024-03-25T18:14:07.795 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.794+0000 7f2456ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:07.795 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:07.794+0000 7f2456ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:08.046 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:08.045+0000 7f2468579640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:08.046 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:08.045+0000 7f2468579640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:08.046 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:08.045+0000 7f2468579640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:14:08.046 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:08.045+0000 7f2468579640 1 -- 172.21.15.66:0/419645173 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f246013f180 con 0x7f246013f710 2024-03-25T18:14:08.076 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:08.075+0000 7f2456ffd640 1 -- 172.21.15.66:0/419645173 <== mon.0 v1:172.21.15.66:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4258808387) 0x7f246013f180 con 0x7f246013f710 2024-03-25T18:14:08.076 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:08.076+0000 7f2456ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:08.076 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:08.076+0000 7f2456ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:08.076 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:14:08.076 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":11,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:13:50.353085+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:50.353085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:08.079 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:08.078+0000 7f2468579640 10 librados: watch_flush enter 2024-03-25T18:14:08.079 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:08.078+0000 7f2468579640 10 librados: watch_flush exit 2024-03-25T18:14:08.079 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:08.078+0000 7f2468579640 20 client.4689.objecter shutdown clearing up homeless session... 2024-03-25T18:14:08.080 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:08.078+0000 7f2468579640 10 client.4689.objecter successfully canceled tick 2024-03-25T18:14:08.080 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:08.078+0000 7f2468579640 1 -- 172.21.15.66:0/419645173 >> v1:172.21.15.193:6800/2275676803 conn(0x7f244406e0f0 legacy=0x7f24440705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:08.080 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:08.078+0000 7f2468579640 10 monclient: shutdown 2024-03-25T18:14:08.080 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:08.078+0000 7f2468579640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:08.080 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:08.078+0000 7f2468579640 1 -- 172.21.15.66:0/419645173 >> v1:172.21.15.66:6789/0 conn(0x7f246013f710 legacy=0x7f24601d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:08.080 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:08.078+0000 7f2468579640 1 -- 172.21.15.66:0/419645173 shutdown_connections 2024-03-25T18:14:08.080 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:08.078+0000 7f2468579640 1 -- 172.21.15.66:0/419645173 >> 172.21.15.66:0/419645173 conn(0x7f24600b7ea0 msgr2=0x7f2460074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:08.080 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:08.078+0000 7f2468579640 1 -- 172.21.15.66:0/419645173 shutdown_connections 2024-03-25T18:14:08.080 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:08.078+0000 7f2468579640 1 -- 172.21.15.66:0/419645173 wait complete. 2024-03-25T18:14:08.080 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:08.079+0000 7f2468579640 1 librados: shutdown 2024-03-25T18:14:08.091 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:14.898 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:15.021 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.020+0000 7f8717ef1640 10 monclient: get_monmap_and_config 2024-03-25T18:14:15.021 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.020+0000 7f8717ef1640 10 monclient: build_initial_monmap 2024-03-25T18:14:15.021 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.020+0000 7f8717ef1640 10 monclient: monmap: 2024-03-25T18:14:15.021 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:14:15.021 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:15.022 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:14:15.020740+0000 2024-03-25T18:14:15.022 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:14:15.020740+0000 2024-03-25T18:14:15.022 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:15.022 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:15.022 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:14:15.022 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:14:15.023 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:14:15.023 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:15.025 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.021+0000 7f8717ef1640 1 Processor -- start 2024-03-25T18:14:15.025 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.021+0000 7f8717ef1640 1 -- start start 2024-03-25T18:14:15.025 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.021+0000 7f8717ef1640 10 monclient: init 2024-03-25T18:14:15.025 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.022+0000 7f8717ef1640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:15.026 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.022+0000 7f8717ef1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:15.026 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.022+0000 7f8717ef1640 10 monclient(hunting): picked mon.noname-a con 0x7f87100a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:14:15.026 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.022+0000 7f8717ef1640 10 monclient(hunting): picked mon.noname-b con 0x7f871013f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:14:15.026 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.022+0000 7f8717ef1640 10 monclient(hunting): picked mon.noname-c con 0x7f87101430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:14:15.026 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.022+0000 7f8717ef1640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8710142520 con 0x7f87100a98f0 2024-03-25T18:14:15.026 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.022+0000 7f8717ef1640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87100b28a0 con 0x7f87101430c0 2024-03-25T18:14:15.026 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.022+0000 7f8717ef1640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87100b2a30 con 0x7f871013f710 2024-03-25T18:14:15.026 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.022+0000 7f8717ef1640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:15.027 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.022+0000 7f8717ef1640 10 monclient(hunting): authenticate will time out at 7604.081659s 2024-03-25T18:14:15.035 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.034+0000 7f871548c640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f871013f710 0x7f87101429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:43672/0 (socket says 172.21.15.66:43672) 2024-03-25T18:14:15.035 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.034+0000 7f871548c640 1 -- 172.21.15.66:0/4121247093 learned_addr learned my addr 172.21.15.66:0/4121247093 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:14:15.052 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.051+0000 7f8714c8b640 1 -- 172.21.15.66:0/4121247093 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3926737152 0 0) 0x7f87100b28a0 con 0x7f87101430c0 2024-03-25T18:14:15.052 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.051+0000 7f8714c8b640 10 monclient(hunting): _negotiate 2024-03-25T18:14:15.052 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.051+0000 7f8714c8b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:15.052 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.051+0000 7f8714c8b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:15.052 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.051+0000 7f8714c8b640 10 monclient(hunting): my global_id is 4687 2024-03-25T18:14:15.052 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.051+0000 7f8714c8b640 1 -- 172.21.15.66:0/4121247093 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86fc003520 con 0x7f87101430c0 2024-03-25T18:14:15.054 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.053+0000 7f8714c8b640 1 -- 172.21.15.66:0/4121247093 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3120233109 0 0) 0x7f8710142520 con 0x7f87100a98f0 2024-03-25T18:14:15.054 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.053+0000 7f8714c8b640 10 monclient(hunting): _negotiate 2024-03-25T18:14:15.054 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.053+0000 7f8714c8b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:15.054 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.053+0000 7f8714c8b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:15.054 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.053+0000 7f8714c8b640 10 monclient(hunting): my global_id is 4692 2024-03-25T18:14:15.054 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.053+0000 7f8714c8b640 1 -- 172.21.15.66:0/4121247093 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87100b28a0 con 0x7f87100a98f0 2024-03-25T18:14:15.060 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.059+0000 7f8714c8b640 1 -- 172.21.15.66:0/4121247093 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3847528683 0 0) 0x7f86fc003520 con 0x7f87101430c0 2024-03-25T18:14:15.060 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.059+0000 7f8714c8b640 1 -- 172.21.15.66:0/4121247093 >> v1:172.21.15.66:6790/0 conn(0x7f871013f710 legacy=0x7f87101429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:15.060 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.059+0000 7f8714c8b640 1 -- 172.21.15.66:0/4121247093 >> v1:172.21.15.66:6789/0 conn(0x7f87100a98f0 legacy=0x7f87100a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:15.060 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.059+0000 7f8714c8b640 10 monclient: _finish_hunting 0 2024-03-25T18:14:15.060 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.059+0000 7f8714c8b640 1 monclient: found mon.noname-c 2024-03-25T18:14:15.060 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.059+0000 7f8714c8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:15.060 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.059+0000 7f8714c8b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:14:15.060 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.059+0000 7f8714c8b640 1 -- 172.21.15.66:0/4121247093 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87100b44c0 con 0x7f87101430c0 2024-03-25T18:14:15.060 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.059+0000 7f8714c8b640 10 monclient: _finish_auth 0 2024-03-25T18:14:15.060 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.059+0000 7f8714c8b640 10 monclient: _check_auth_tickets 2024-03-25T18:14:15.060 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.059+0000 7f8714c8b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:15.061 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.059+0000 7f8714c8b640 1 -- 172.21.15.66:0/4121247093 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8708001a10 con 0x7f87101430c0 2024-03-25T18:14:15.061 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.059+0000 7f8714c8b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:15.061 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.059+0000 7f8714c8b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:15.061 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.059+0000 7f8714c8b640 10 monclient: dump: 2024-03-25T18:14:15.061 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:15.061 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:15.061 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:15.061 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:15.061 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:15.061 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:15.061 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:15.061 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:15.061 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:15.062 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:15.062 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.059+0000 7f8717ef1640 5 monclient: authenticate success, global_id 4687 2024-03-25T18:14:15.062 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.059+0000 7f8717ef1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:15.071 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.071+0000 7f8714c8b640 1 -- 172.21.15.66:0/4121247093 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8708001bf0 con 0x7f87101430c0 2024-03-25T18:14:15.071 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.071+0000 7f8714c8b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:15.071 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.071+0000 7f8714c8b640 1 -- 172.21.15.66:0/4121247093 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8708002000 con 0x7f87101430c0 2024-03-25T18:14:15.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.071+0000 7f8717ef1640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:15.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.071+0000 7f8717ef1640 4 set_mon_vals no callback set 2024-03-25T18:14:15.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.071+0000 7f8714c8b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:15.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.071+0000 7f8714c8b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:15.072 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.071+0000 7f8714c8b640 10 monclient: dump: 2024-03-25T18:14:15.072 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:15.072 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:15.072 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:15.072 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:15.072 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:15.072 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:15.072 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:15.072 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:15.072 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:15.072 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:15.074 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.072+0000 7f8717ef1640 10 monclient: shutdown 2024-03-25T18:14:15.074 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.072+0000 7f8717ef1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:15.074 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.072+0000 7f8717ef1640 1 -- 172.21.15.66:0/4121247093 >> v1:172.21.15.193:6789/0 conn(0x7f87101430c0 legacy=0x7f871014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:15.074 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.072+0000 7f8717ef1640 1 -- 172.21.15.66:0/4121247093 shutdown_connections 2024-03-25T18:14:15.074 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.072+0000 7f8717ef1640 1 -- 172.21.15.66:0/4121247093 >> 172.21.15.66:0/4121247093 conn(0x7f87100b7ea0 msgr2=0x7f87100b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:15.074 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.072+0000 7f8717ef1640 1 -- 172.21.15.66:0/4121247093 shutdown_connections 2024-03-25T18:14:15.074 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.072+0000 7f8717ef1640 1 -- 172.21.15.66:0/4121247093 wait complete. 2024-03-25T18:14:15.074 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.073+0000 7f8717ef1640 10 monclient: build_initial_monmap 2024-03-25T18:14:15.074 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.073+0000 7f8717ef1640 10 monclient: monmap: 2024-03-25T18:14:15.075 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:14:15.075 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:15.075 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:14:15.075 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:14:15.075 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:15.075 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:14:15.075 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:14:15.075 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:14:15.075 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:14:15.075 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:15.075 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.073+0000 7f8717ef1640 1 librados: starting msgr at 2024-03-25T18:14:15.075 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.073+0000 7f8717ef1640 1 librados: starting objecter 2024-03-25T18:14:15.075 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.073+0000 7f8717ef1640 1 Processor -- start 2024-03-25T18:14:15.075 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.073+0000 7f8717ef1640 1 -- start start 2024-03-25T18:14:15.075 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.073+0000 7f8717ef1640 1 librados: setting wanted keys 2024-03-25T18:14:15.076 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.073+0000 7f8717ef1640 1 librados: calling monclient init 2024-03-25T18:14:15.076 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.073+0000 7f8717ef1640 10 monclient: init 2024-03-25T18:14:15.076 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.074+0000 7f8717ef1640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:15.077 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.074+0000 7f8717ef1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:15.077 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.074+0000 7f8717ef1640 10 monclient(hunting): picked mon.noname-a con 0x7f87100a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:14:15.077 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.074+0000 7f8717ef1640 10 monclient(hunting): picked mon.noname-b con 0x7f871013f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:14:15.077 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.074+0000 7f8717ef1640 10 monclient(hunting): picked mon.noname-c con 0x7f87101430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:14:15.077 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.074+0000 7f8717ef1640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87101d7590 con 0x7f87100a98f0 2024-03-25T18:14:15.077 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.074+0000 7f8717ef1640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87101d51f0 con 0x7f871013f710 2024-03-25T18:14:15.077 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.074+0000 7f8717ef1640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87101d8410 con 0x7f87101430c0 2024-03-25T18:14:15.077 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.074+0000 7f8717ef1640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:15.077 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.074+0000 7f8717ef1640 10 monclient(hunting): authenticate will time out at 7604.133660s 2024-03-25T18:14:15.081 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.081+0000 7f8715c8d640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f87100a98f0 0x7f87101d7d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:47480/0 (socket says 172.21.15.66:47480) 2024-03-25T18:14:15.081 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.081+0000 7f8715c8d640 1 -- 172.21.15.66:0/2186089254 learned_addr learned my addr 172.21.15.66:0/2186089254 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:14:15.083 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.083+0000 7f87067fc640 10 client.?.objecter ms_handle_connect 0x7f87100a98f0 2024-03-25T18:14:15.083 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.083+0000 7f87067fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:15.084 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.084+0000 7f87067fc640 10 client.?.objecter ms_handle_connect 0x7f871013f710 2024-03-25T18:14:15.085 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.084+0000 7f87067fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:15.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.088+0000 7f87067fc640 10 client.?.objecter ms_handle_connect 0x7f87101430c0 2024-03-25T18:14:15.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.088+0000 7f87067fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:15.104 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.103+0000 7f87067fc640 1 -- 172.21.15.66:0/2186089254 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1207807716 0 0) 0x7f87101d51f0 con 0x7f871013f710 2024-03-25T18:14:15.104 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.103+0000 7f87067fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:15.104 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.103+0000 7f87067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:15.104 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.103+0000 7f87067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:15.104 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.103+0000 7f87067fc640 10 monclient(hunting): my global_id is 4690 2024-03-25T18:14:15.104 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.103+0000 7f87067fc640 1 -- 172.21.15.66:0/2186089254 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86f4004eb0 con 0x7f871013f710 2024-03-25T18:14:15.107 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.107+0000 7f87067fc640 1 -- 172.21.15.66:0/2186089254 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1323037924 0 0) 0x7f87101d7590 con 0x7f87100a98f0 2024-03-25T18:14:15.107 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.107+0000 7f87067fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:15.107 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.107+0000 7f87067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:15.107 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.107+0000 7f87067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:15.107 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.107+0000 7f87067fc640 10 monclient(hunting): my global_id is 4695 2024-03-25T18:14:15.107 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.107+0000 7f87067fc640 1 -- 172.21.15.66:0/2186089254 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87101d51f0 con 0x7f87100a98f0 2024-03-25T18:14:15.113 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.113+0000 7f87067fc640 1 -- 172.21.15.66:0/2186089254 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2246529717 0 0) 0x7f87101d8410 con 0x7f87101430c0 2024-03-25T18:14:15.113 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.113+0000 7f87067fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:15.114 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.113+0000 7f87067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:15.114 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.113+0000 7f87067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:15.114 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.113+0000 7f87067fc640 10 monclient(hunting): my global_id is 4688 2024-03-25T18:14:15.114 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.113+0000 7f87067fc640 1 -- 172.21.15.66:0/2186089254 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87101d7590 con 0x7f87101430c0 2024-03-25T18:14:15.128 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.127+0000 7f87067fc640 1 -- 172.21.15.66:0/2186089254 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3908403021 0 0) 0x7f86f4004eb0 con 0x7f871013f710 2024-03-25T18:14:15.128 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.127+0000 7f87067fc640 1 -- 172.21.15.66:0/2186089254 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87101d8410 con 0x7f871013f710 2024-03-25T18:14:15.128 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.127+0000 7f87067fc640 1 -- 172.21.15.66:0/2186089254 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f87000018a0 con 0x7f871013f710 2024-03-25T18:14:15.128 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.127+0000 7f87067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:15.128 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.127+0000 7f87067fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:15.128 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.127+0000 7f87067fc640 10 monclient(hunting): dump: 2024-03-25T18:14:15.128 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:15.128 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:15.128 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:15.128 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:15.129 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:15.129 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:15.129 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:15.129 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:15.129 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:15.129 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:15.129 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.127+0000 7f87067fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:15.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.136+0000 7f87067fc640 1 -- 172.21.15.66:0/2186089254 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2818176067 0 0) 0x7f87101d51f0 con 0x7f87100a98f0 2024-03-25T18:14:15.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.137+0000 7f87067fc640 1 -- 172.21.15.66:0/2186089254 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86f4004eb0 con 0x7f87100a98f0 2024-03-25T18:14:15.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.137+0000 7f87067fc640 1 -- 172.21.15.66:0/2186089254 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f870c0019f0 con 0x7f87100a98f0 2024-03-25T18:14:15.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.137+0000 7f87067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:15.137 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.137+0000 7f87067fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:15.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.137+0000 7f87067fc640 10 monclient(hunting): dump: 2024-03-25T18:14:15.138 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:15.138 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:15.138 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:15.138 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:15.138 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:15.138 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:15.138 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:15.138 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:15.138 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:15.138 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:15.140 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.139+0000 7f87067fc640 1 -- 172.21.15.66:0/2186089254 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 89977217 0 0) 0x7f87101d7590 con 0x7f87101430c0 2024-03-25T18:14:15.140 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.139+0000 7f87067fc640 1 -- 172.21.15.66:0/2186089254 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87101d51f0 con 0x7f87101430c0 2024-03-25T18:14:15.140 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.139+0000 7f87067fc640 1 -- 172.21.15.66:0/2186089254 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8708001f00 con 0x7f87101430c0 2024-03-25T18:14:15.140 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.139+0000 7f87067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:15.140 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.139+0000 7f87067fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:15.141 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.139+0000 7f87067fc640 10 monclient(hunting): dump: 2024-03-25T18:14:15.141 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:15.141 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:15.141 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:15.141 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:15.141 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:15.141 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:15.141 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:15.141 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:15.141 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:15.141 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:15.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.147+0000 7f87067fc640 1 -- 172.21.15.66:0/2186089254 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 744264533 0 0) 0x7f87101d8410 con 0x7f871013f710 2024-03-25T18:14:15.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.147+0000 7f87067fc640 1 -- 172.21.15.66:0/2186089254 >> v1:172.21.15.66:6790/0 conn(0x7f87101430c0 legacy=0x7f87101e1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:15.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.147+0000 7f87067fc640 1 -- 172.21.15.66:0/2186089254 >> v1:172.21.15.66:6789/0 conn(0x7f87100a98f0 legacy=0x7f87101d7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:15.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.147+0000 7f87067fc640 10 monclient: _finish_hunting 0 2024-03-25T18:14:15.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.147+0000 7f87067fc640 1 monclient: found mon.b 2024-03-25T18:14:15.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.147+0000 7f87067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:15.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.147+0000 7f87067fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:14:15.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.147+0000 7f87067fc640 1 -- 172.21.15.66:0/2186089254 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87101d77e0 con 0x7f871013f710 2024-03-25T18:14:15.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.147+0000 7f87067fc640 10 monclient: _finish_auth 0 2024-03-25T18:14:15.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.147+0000 7f87067fc640 10 monclient: _check_auth_tickets 2024-03-25T18:14:15.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.147+0000 7f87067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:15.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.147+0000 7f8717ef1640 5 monclient: authenticate success, global_id 4690 2024-03-25T18:14:15.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.147+0000 7f8717ef1640 10 monclient: _renew_subs 2024-03-25T18:14:15.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.147+0000 7f8717ef1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:14:15.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.147+0000 7f8717ef1640 1 -- 172.21.15.66:0/2186089254 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8710138ce0 con 0x7f871013f710 2024-03-25T18:14:15.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.147+0000 7f8717ef1640 10 client.4690.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:15.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.147+0000 7f8717ef1640 10 monclient: _renew_subs 2024-03-25T18:14:15.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.147+0000 7f8717ef1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:14:15.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.147+0000 7f8717ef1640 1 -- 172.21.15.66:0/2186089254 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8710137ff0 con 0x7f871013f710 2024-03-25T18:14:15.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.147+0000 7f8717ef1640 1 librados: init done 2024-03-25T18:14:15.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.148+0000 7f8717ef1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:15.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.148+0000 7f8717ef1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:15.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.148+0000 7f8717ef1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:14:15.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.148+0000 7f8717ef1640 1 -- 172.21.15.66:0/2186089254 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87100627b0 con 0x7f871013f710 2024-03-25T18:14:15.442 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.441+0000 7f87067fc640 1 -- 172.21.15.66:0/2186089254 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8700001d10 con 0x7f871013f710 2024-03-25T18:14:15.442 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.441+0000 7f87067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:15.442 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.441+0000 7f87067fc640 1 -- 172.21.15.66:0/2186089254 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8700004a00 con 0x7f871013f710 2024-03-25T18:14:15.442 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.441+0000 7f87067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:15.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.441+0000 7f87067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:15.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.441+0000 7f87067fc640 10 monclient: dump: 2024-03-25T18:14:15.443 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:15.443 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:15.443 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:15.443 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:15.443 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:15.443 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:15.443 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:15.443 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:15.443 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:15.443 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:15.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.441+0000 7f8707fff640 4 set_mon_vals no callback set 2024-03-25T18:14:15.444 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.442+0000 7f87067fc640 1 -- 172.21.15.66:0/2186089254 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f870001aa50 con 0x7f871013f710 2024-03-25T18:14:15.444 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.443+0000 7f8715c8d640 10 monclient: get_auth_request con 0x7f86f406e0b0 auth_method 0 2024-03-25T18:14:15.444 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.443+0000 7f87067fc640 10 client.4690.objecter ms_handle_connect 0x7f86f406e0b0 2024-03-25T18:14:15.449 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.449+0000 7f87067fc640 1 -- 172.21.15.66:0/2186089254 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4152745027 0 0) 0x7f87000547a0 con 0x7f871013f710 2024-03-25T18:14:15.449 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.449+0000 7f87067fc640 10 client.4690.objecter ms_dispatch 0x7f8710000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:14:15.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.449+0000 7f87067fc640 3 client.4690.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:14:15.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.449+0000 7f87067fc640 3 client.4690.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:14:15.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.449+0000 7f87067fc640 20 client.4690.objecter dump_active .. 0 homeless 2024-03-25T18:14:15.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.449+0000 7f87067fc640 1 -- 172.21.15.66:0/2186089254 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f87000500e0 con 0x7f871013f710 2024-03-25T18:14:15.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.449+0000 7f87067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:15.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.449+0000 7f87067fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:15.524 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:10:46.504 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:15.585 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:10:46.565 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:15.620 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:10:46.600 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:15.653 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:10:46.611 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:15.655 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:10:46.635 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:15.660 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:10:46.641 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:15.662 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:10:46.621 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:15.679 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.679+0000 7f8717ef1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:15.679 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.679+0000 7f8717ef1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:15.680 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.679+0000 7f8717ef1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:14:15.680 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.679+0000 7f8717ef1640 1 -- 172.21.15.66:0/2186089254 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87101d7970 con 0x7f871013f710 2024-03-25T18:14:15.687 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:10:46.668 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:15.697 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:10:46.656 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:15.781 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.781+0000 7f87067fc640 1 -- 172.21.15.66:0/2186089254 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4258808387) 0x7f87000521e0 con 0x7f871013f710 2024-03-25T18:14:15.781 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.781+0000 7f87067fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:15.781 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.781+0000 7f87067fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:15.782 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:14:15.782 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":11,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:13:50.353085+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:50.353085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:15.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.783+0000 7f8717ef1640 10 librados: watch_flush enter 2024-03-25T18:14:15.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.783+0000 7f8717ef1640 10 librados: watch_flush exit 2024-03-25T18:14:15.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.783+0000 7f8717ef1640 20 client.4690.objecter shutdown clearing up homeless session... 2024-03-25T18:14:15.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.783+0000 7f8717ef1640 10 client.4690.objecter successfully canceled tick 2024-03-25T18:14:15.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.783+0000 7f8717ef1640 1 -- 172.21.15.66:0/2186089254 >> v1:172.21.15.193:6800/2275676803 conn(0x7f86f406e0b0 legacy=0x7f86f4070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:15.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.783+0000 7f8717ef1640 10 monclient: shutdown 2024-03-25T18:14:15.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.783+0000 7f8717ef1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:15.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.783+0000 7f8717ef1640 1 -- 172.21.15.66:0/2186089254 >> v1:172.21.15.193:6789/0 conn(0x7f871013f710 legacy=0x7f87101385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:15.785 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.783+0000 7f8717ef1640 1 -- 172.21.15.66:0/2186089254 shutdown_connections 2024-03-25T18:14:15.785 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.783+0000 7f8717ef1640 1 -- 172.21.15.66:0/2186089254 >> 172.21.15.66:0/2186089254 conn(0x7f87100b7ea0 msgr2=0x7f8710074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:15.785 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.783+0000 7f8717ef1640 1 -- 172.21.15.66:0/2186089254 shutdown_connections 2024-03-25T18:14:15.785 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.783+0000 7f8717ef1640 1 -- 172.21.15.66:0/2186089254 wait complete. 2024-03-25T18:14:15.785 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:15.783+0000 7f8717ef1640 1 librados: shutdown 2024-03-25T18:14:15.795 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:22.601 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:22.719 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.718+0000 7f125fc25640 10 monclient: get_monmap_and_config 2024-03-25T18:14:22.719 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.718+0000 7f125fc25640 10 monclient: build_initial_monmap 2024-03-25T18:14:22.719 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.718+0000 7f125fc25640 10 monclient: monmap: 2024-03-25T18:14:22.719 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:14:22.719 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:22.719 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:14:22.719428+0000 2024-03-25T18:14:22.719 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:14:22.719428+0000 2024-03-25T18:14:22.719 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:22.719 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:22.719 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:14:22.720 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:14:22.720 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:14:22.720 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:22.720 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.719+0000 7f125fc25640 1 Processor -- start 2024-03-25T18:14:22.720 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.719+0000 7f125fc25640 1 -- start start 2024-03-25T18:14:22.720 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.719+0000 7f125fc25640 10 monclient: init 2024-03-25T18:14:22.721 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.720+0000 7f125fc25640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:22.721 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.720+0000 7f125fc25640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:22.721 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.720+0000 7f125fc25640 10 monclient(hunting): picked mon.noname-b con 0x7f12580b4d60 addr v1:172.21.15.66:6790/0 2024-03-25T18:14:22.721 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.720+0000 7f125fc25640 10 monclient(hunting): picked mon.noname-c con 0x7f12580b1730 addr v1:172.21.15.193:6789/0 2024-03-25T18:14:22.721 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.720+0000 7f125fc25640 10 monclient(hunting): picked mon.noname-a con 0x7f125814be90 addr v1:172.21.15.66:6789/0 2024-03-25T18:14:22.722 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.720+0000 7f125fc25640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12580af940 con 0x7f125814be90 2024-03-25T18:14:22.722 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.720+0000 7f125fc25640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12580b0a70 con 0x7f12580b1730 2024-03-25T18:14:22.722 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.720+0000 7f125fc25640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12580b0d50 con 0x7f12580b4d60 2024-03-25T18:14:22.722 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.720+0000 7f125fc25640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:22.722 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.720+0000 7f125fc25640 10 monclient(hunting): authenticate will time out at 7611.779804s 2024-03-25T18:14:22.729 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.729+0000 7f125e1c2640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f125814be90 0x7f125814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:51840/0 (socket says 172.21.15.66:51840) 2024-03-25T18:14:22.730 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.729+0000 7f125e1c2640 1 -- 172.21.15.66:0/3963565492 learned_addr learned my addr 172.21.15.66:0/3963565492 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:14:22.743 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.742+0000 7f125c9bf640 1 -- 172.21.15.66:0/3963565492 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502478139 0 0) 0x7f12580b0d50 con 0x7f12580b4d60 2024-03-25T18:14:22.743 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.742+0000 7f125c9bf640 10 monclient(hunting): _negotiate 2024-03-25T18:14:22.743 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.742+0000 7f125c9bf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:22.743 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.742+0000 7f125c9bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:22.743 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.742+0000 7f125c9bf640 10 monclient(hunting): my global_id is 4691 2024-03-25T18:14:22.743 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.742+0000 7f125c9bf640 1 -- 172.21.15.66:0/3963565492 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1244003520 con 0x7f12580b4d60 2024-03-25T18:14:22.743 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.742+0000 7f125c9bf640 1 -- 172.21.15.66:0/3963565492 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338962642 0 0) 0x7f12580b0a70 con 0x7f12580b1730 2024-03-25T18:14:22.743 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.742+0000 7f125c9bf640 10 monclient(hunting): _negotiate 2024-03-25T18:14:22.743 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.742+0000 7f125c9bf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:22.743 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.742+0000 7f125c9bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:22.743 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.742+0000 7f125c9bf640 10 monclient(hunting): my global_id is 4693 2024-03-25T18:14:22.743 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.742+0000 7f125c9bf640 1 -- 172.21.15.66:0/3963565492 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12580b0d50 con 0x7f12580b1730 2024-03-25T18:14:22.745 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.744+0000 7f125c9bf640 1 -- 172.21.15.66:0/3963565492 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2221840370 0 0) 0x7f12580af940 con 0x7f125814be90 2024-03-25T18:14:22.745 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.744+0000 7f125c9bf640 10 monclient(hunting): _negotiate 2024-03-25T18:14:22.745 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.744+0000 7f125c9bf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:22.745 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.744+0000 7f125c9bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:22.745 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.744+0000 7f125c9bf640 10 monclient(hunting): my global_id is 4698 2024-03-25T18:14:22.745 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.744+0000 7f125c9bf640 1 -- 172.21.15.66:0/3963565492 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12580b0a70 con 0x7f125814be90 2024-03-25T18:14:22.751 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.751+0000 7f125c9bf640 1 -- 172.21.15.66:0/3963565492 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3667496863 0 0) 0x7f1244003520 con 0x7f12580b4d60 2024-03-25T18:14:22.751 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.751+0000 7f125c9bf640 1 -- 172.21.15.66:0/3963565492 >> v1:172.21.15.193:6789/0 conn(0x7f12580b1730 legacy=0x7f12580b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:22.751 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.751+0000 7f125c9bf640 1 -- 172.21.15.66:0/3963565492 >> v1:172.21.15.66:6789/0 conn(0x7f125814be90 legacy=0x7f125814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:22.751 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.751+0000 7f125c9bf640 10 monclient: _finish_hunting 0 2024-03-25T18:14:22.752 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.751+0000 7f125c9bf640 1 monclient: found mon.noname-b 2024-03-25T18:14:22.752 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.751+0000 7f125c9bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:22.752 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.751+0000 7f125c9bf640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.66:6790/0 2024-03-25T18:14:22.752 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.751+0000 7f125c9bf640 1 -- 172.21.15.66:0/3963565492 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12580b1030 con 0x7f12580b4d60 2024-03-25T18:14:22.752 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.751+0000 7f125c9bf640 10 monclient: _finish_auth 0 2024-03-25T18:14:22.752 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.751+0000 7f125c9bf640 10 monclient: _check_auth_tickets 2024-03-25T18:14:22.752 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.751+0000 7f125c9bf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:22.752 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.751+0000 7f125c9bf640 1 -- 172.21.15.66:0/3963565492 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f1254001690 con 0x7f12580b4d60 2024-03-25T18:14:22.752 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.751+0000 7f125c9bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:22.752 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.751+0000 7f125c9bf640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:22.752 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.751+0000 7f125c9bf640 10 monclient: dump: 2024-03-25T18:14:22.752 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:22.752 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:22.752 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:22.752 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:22.752 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:22.753 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:22.753 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:22.753 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:22.753 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:22.753 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:22.753 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.751+0000 7f125fc25640 5 monclient: authenticate success, global_id 4691 2024-03-25T18:14:22.753 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.751+0000 7f125fc25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:22.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.757+0000 7f125c9bf640 1 -- 172.21.15.66:0/3963565492 <== mon.2 v1:172.21.15.66:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12540017f0 con 0x7f12580b4d60 2024-03-25T18:14:22.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.757+0000 7f125c9bf640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:22.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.757+0000 7f125c9bf640 1 -- 172.21.15.66:0/3963565492 <== mon.2 v1:172.21.15.66:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f1254001c00 con 0x7f12580b4d60 2024-03-25T18:14:22.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.757+0000 7f125fc25640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:22.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.757+0000 7f125fc25640 4 set_mon_vals no callback set 2024-03-25T18:14:22.758 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.757+0000 7f125c9bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:22.759 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.757+0000 7f125c9bf640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:22.759 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.757+0000 7f125c9bf640 10 monclient: dump: 2024-03-25T18:14:22.759 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:22.759 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:22.759 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:22.759 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:22.759 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:22.759 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:22.759 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:22.759 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:22.759 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:22.759 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:22.761 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.758+0000 7f125fc25640 10 monclient: shutdown 2024-03-25T18:14:22.761 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.758+0000 7f125fc25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:22.761 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.758+0000 7f125fc25640 1 -- 172.21.15.66:0/3963565492 >> v1:172.21.15.66:6790/0 conn(0x7f12580b4d60 legacy=0x7f125813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:22.761 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.758+0000 7f125fc25640 1 -- 172.21.15.66:0/3963565492 shutdown_connections 2024-03-25T18:14:22.761 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.758+0000 7f125fc25640 1 -- 172.21.15.66:0/3963565492 >> 172.21.15.66:0/3963565492 conn(0x7f12580617a0 msgr2=0x7f1258061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:22.761 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.759+0000 7f125fc25640 1 -- 172.21.15.66:0/3963565492 shutdown_connections 2024-03-25T18:14:22.761 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.759+0000 7f125fc25640 1 -- 172.21.15.66:0/3963565492 wait complete. 2024-03-25T18:14:22.761 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.759+0000 7f125fc25640 10 monclient: build_initial_monmap 2024-03-25T18:14:22.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.759+0000 7f125fc25640 10 monclient: monmap: 2024-03-25T18:14:22.762 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:14:22.762 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:22.762 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:14:22.762 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:14:22.762 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:22.762 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:14:22.762 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:14:22.762 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:14:22.762 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:14:22.762 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:22.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.760+0000 7f125fc25640 1 librados: starting msgr at 2024-03-25T18:14:22.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.760+0000 7f125fc25640 1 librados: starting objecter 2024-03-25T18:14:22.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.760+0000 7f125fc25640 1 Processor -- start 2024-03-25T18:14:22.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.760+0000 7f125fc25640 1 -- start start 2024-03-25T18:14:22.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.760+0000 7f125fc25640 1 librados: setting wanted keys 2024-03-25T18:14:22.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.760+0000 7f125fc25640 1 librados: calling monclient init 2024-03-25T18:14:22.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.760+0000 7f125fc25640 10 monclient: init 2024-03-25T18:14:22.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.760+0000 7f125fc25640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:22.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.760+0000 7f125fc25640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:22.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.760+0000 7f125fc25640 10 monclient(hunting): picked mon.noname-b con 0x7f12580b1730 addr v1:172.21.15.193:6789/0 2024-03-25T18:14:22.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.760+0000 7f125fc25640 10 monclient(hunting): picked mon.noname-a con 0x7f12580b4d60 addr v1:172.21.15.66:6789/0 2024-03-25T18:14:22.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.760+0000 7f125fc25640 10 monclient(hunting): picked mon.noname-c con 0x7f125814be90 addr v1:172.21.15.66:6790/0 2024-03-25T18:14:22.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.760+0000 7f125fc25640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1258136870 con 0x7f12580b4d60 2024-03-25T18:14:22.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.760+0000 7f125fc25640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1258136350 con 0x7f12580b1730 2024-03-25T18:14:22.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.760+0000 7f125fc25640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1258136fa0 con 0x7f125814be90 2024-03-25T18:14:22.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.760+0000 7f125fc25640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:22.763 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.760+0000 7f125fc25640 10 monclient(hunting): authenticate will time out at 7611.819999s 2024-03-25T18:14:22.774 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.773+0000 7f125d9c1640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f12580b1730 0x7f125812fcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:42326/0 (socket says 172.21.15.66:42326) 2024-03-25T18:14:22.774 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.773+0000 7f125d9c1640 1 -- 172.21.15.66:0/2041761979 learned_addr learned my addr 172.21.15.66:0/2041761979 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:14:22.777 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.774+0000 7f124e7fc640 10 client.?.objecter ms_handle_connect 0x7f12580b1730 2024-03-25T18:14:22.777 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.774+0000 7f124e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:22.779 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.778+0000 7f124e7fc640 10 client.?.objecter ms_handle_connect 0x7f125814be90 2024-03-25T18:14:22.779 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.778+0000 7f124e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:22.782 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.781+0000 7f124e7fc640 10 client.?.objecter ms_handle_connect 0x7f12580b4d60 2024-03-25T18:14:22.782 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.781+0000 7f124e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:22.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.783+0000 7f124e7fc640 1 -- 172.21.15.66:0/2041761979 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1854546970 0 0) 0x7f1258136fa0 con 0x7f125814be90 2024-03-25T18:14:22.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.784+0000 7f124e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:22.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.784+0000 7f124e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:22.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.784+0000 7f124e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:22.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.784+0000 7f124e7fc640 10 monclient(hunting): my global_id is 4694 2024-03-25T18:14:22.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.784+0000 7f124e7fc640 1 -- 172.21.15.66:0/2041761979 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f123c004eb0 con 0x7f125814be90 2024-03-25T18:14:22.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.789+0000 7f124e7fc640 1 -- 172.21.15.66:0/2041761979 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2291171527 0 0) 0x7f1258136350 con 0x7f12580b1730 2024-03-25T18:14:22.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.789+0000 7f124e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:22.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.789+0000 7f124e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:22.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.789+0000 7f124e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:22.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.789+0000 7f124e7fc640 10 monclient(hunting): my global_id is 4696 2024-03-25T18:14:22.790 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.789+0000 7f124e7fc640 1 -- 172.21.15.66:0/2041761979 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1258136fa0 con 0x7f12580b1730 2024-03-25T18:14:22.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.791+0000 7f124e7fc640 1 -- 172.21.15.66:0/2041761979 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3979194647 0 0) 0x7f123c004eb0 con 0x7f125814be90 2024-03-25T18:14:22.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.791+0000 7f124e7fc640 1 -- 172.21.15.66:0/2041761979 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1258136350 con 0x7f125814be90 2024-03-25T18:14:22.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.791+0000 7f124e7fc640 1 -- 172.21.15.66:0/2041761979 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f1250001bd0 con 0x7f125814be90 2024-03-25T18:14:22.791 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.791+0000 7f124e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:22.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.791+0000 7f124e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:22.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.791+0000 7f124e7fc640 10 monclient(hunting): dump: 2024-03-25T18:14:22.792 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:22.792 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:22.792 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:22.792 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:22.792 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:22.792 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:22.792 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:22.792 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:22.792 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:22.792 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:22.792 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.791+0000 7f124e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:22.793 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.792+0000 7f124e7fc640 1 -- 172.21.15.66:0/2041761979 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1848739523 0 0) 0x7f1258136870 con 0x7f12580b4d60 2024-03-25T18:14:22.793 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.792+0000 7f124e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:22.793 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.792+0000 7f124e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:22.793 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.792+0000 7f124e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:22.793 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.792+0000 7f124e7fc640 10 monclient(hunting): my global_id is 4701 2024-03-25T18:14:22.793 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.792+0000 7f124e7fc640 1 -- 172.21.15.66:0/2041761979 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f123c004eb0 con 0x7f12580b4d60 2024-03-25T18:14:22.810 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.809+0000 7f124e7fc640 1 -- 172.21.15.66:0/2041761979 <== mon.2 v1:172.21.15.66:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2060950041 0 0) 0x7f1258136350 con 0x7f125814be90 2024-03-25T18:14:22.810 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.809+0000 7f124e7fc640 1 -- 172.21.15.66:0/2041761979 >> v1:172.21.15.193:6789/0 conn(0x7f12580b1730 legacy=0x7f125812fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:22.810 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.809+0000 7f124e7fc640 1 -- 172.21.15.66:0/2041761979 >> v1:172.21.15.66:6789/0 conn(0x7f12580b4d60 legacy=0x7f12581dac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:22.810 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.809+0000 7f124e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:14:22.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.809+0000 7f124e7fc640 1 monclient: found mon.c 2024-03-25T18:14:22.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.809+0000 7f124e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:22.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.809+0000 7f124e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:14:22.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.809+0000 7f124e7fc640 1 -- 172.21.15.66:0/2041761979 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1258138620 con 0x7f125814be90 2024-03-25T18:14:22.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.809+0000 7f124e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:14:22.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.809+0000 7f124e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:14:22.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.809+0000 7f124e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:22.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.809+0000 7f125fc25640 5 monclient: authenticate success, global_id 4694 2024-03-25T18:14:22.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.809+0000 7f125fc25640 10 monclient: _renew_subs 2024-03-25T18:14:22.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.809+0000 7f125fc25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:14:22.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.809+0000 7f125fc25640 1 -- 172.21.15.66:0/2041761979 --> v1:172.21.15.66:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12581324e0 con 0x7f125814be90 2024-03-25T18:14:22.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.809+0000 7f125fc25640 10 client.4694.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:22.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.809+0000 7f125fc25640 10 monclient: _renew_subs 2024-03-25T18:14:22.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.809+0000 7f125fc25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:14:22.811 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.809+0000 7f125fc25640 1 -- 172.21.15.66:0/2041761979 --> v1:172.21.15.66:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1258135f10 con 0x7f125814be90 2024-03-25T18:14:22.812 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.809+0000 7f125fc25640 1 librados: init done 2024-03-25T18:14:22.812 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.811+0000 7f125fc25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:22.812 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.811+0000 7f125fc25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:22.812 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.811+0000 7f125fc25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:14:22.812 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:22.811+0000 7f125fc25640 1 -- 172.21.15.66:0/2041761979 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12581361c0 con 0x7f125814be90 2024-03-25T18:14:23.141 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.140+0000 7f124e7fc640 1 -- 172.21.15.66:0/2041761979 <== mon.2 v1:172.21.15.66:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1250001fe0 con 0x7f125814be90 2024-03-25T18:14:23.141 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.140+0000 7f124e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:23.141 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.140+0000 7f124e7fc640 1 -- 172.21.15.66:0/2041761979 <== mon.2 v1:172.21.15.66:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f1250004d00 con 0x7f125814be90 2024-03-25T18:14:23.141 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.140+0000 7f124e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:23.141 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.141+0000 7f124e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:23.142 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.141+0000 7f124e7fc640 10 monclient: dump: 2024-03-25T18:14:23.142 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:23.142 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:23.142 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:23.142 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:23.142 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:23.142 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:23.142 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:23.142 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:23.142 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:23.142 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:23.142 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.141+0000 7f124ffff640 4 set_mon_vals no callback set 2024-03-25T18:14:23.143 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.142+0000 7f124e7fc640 1 -- 172.21.15.66:0/2041761979 <== mon.2 v1:172.21.15.66:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f1250004f20 con 0x7f125814be90 2024-03-25T18:14:23.143 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.142+0000 7f125d9c1640 10 monclient: get_auth_request con 0x7f123c06de00 auth_method 0 2024-03-25T18:14:23.143 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.143+0000 7f124e7fc640 10 client.4694.objecter ms_handle_connect 0x7f123c06de00 2024-03-25T18:14:23.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.149+0000 7f124e7fc640 1 -- 172.21.15.66:0/2041761979 <== mon.2 v1:172.21.15.66:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4152745027 0 0) 0x7f1250054980 con 0x7f125814be90 2024-03-25T18:14:23.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.149+0000 7f124e7fc640 10 client.4694.objecter ms_dispatch 0x7f1258000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:14:23.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.149+0000 7f124e7fc640 3 client.4694.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:14:23.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.149+0000 7f124e7fc640 3 client.4694.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:14:23.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.149+0000 7f124e7fc640 20 client.4694.objecter dump_active .. 0 homeless 2024-03-25T18:14:23.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.149+0000 7f124e7fc640 1 -- 172.21.15.66:0/2041761979 <== mon.2 v1:172.21.15.66:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1250050490 con 0x7f125814be90 2024-03-25T18:14:23.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.149+0000 7f124e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:23.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.149+0000 7f124e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:23.409 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.408+0000 7f125fc25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:23.410 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.408+0000 7f125fc25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:23.410 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.408+0000 7f125fc25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.66:6790/0 2024-03-25T18:14:23.410 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.408+0000 7f125fc25640 1 -- 172.21.15.66:0/2041761979 --> v1:172.21.15.66:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1258132880 con 0x7f125814be90 2024-03-25T18:14:23.442 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.442+0000 7f124e7fc640 1 -- 172.21.15.66:0/2041761979 <== mon.2 v1:172.21.15.66:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4258808387) 0x7f12500523c0 con 0x7f125814be90 2024-03-25T18:14:23.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.442+0000 7f124e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:23.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.442+0000 7f124e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:23.443 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:14:23.443 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":11,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:13:50.353085+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:50.353085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:23.445 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.445+0000 7f125fc25640 10 librados: watch_flush enter 2024-03-25T18:14:23.446 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.445+0000 7f125fc25640 10 librados: watch_flush exit 2024-03-25T18:14:23.446 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.445+0000 7f125fc25640 20 client.4694.objecter shutdown clearing up homeless session... 2024-03-25T18:14:23.446 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.445+0000 7f125fc25640 10 client.4694.objecter successfully canceled tick 2024-03-25T18:14:23.446 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.445+0000 7f125fc25640 1 -- 172.21.15.66:0/2041761979 >> v1:172.21.15.193:6800/2275676803 conn(0x7f123c06de00 legacy=0x7f123c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:23.446 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.445+0000 7f125fc25640 10 monclient: shutdown 2024-03-25T18:14:23.446 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.445+0000 7f125fc25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:23.446 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.445+0000 7f125fc25640 1 -- 172.21.15.66:0/2041761979 >> v1:172.21.15.66:6790/0 conn(0x7f125814be90 legacy=0x7f12581e2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:23.446 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.445+0000 7f125fc25640 1 -- 172.21.15.66:0/2041761979 shutdown_connections 2024-03-25T18:14:23.446 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.445+0000 7f125fc25640 1 -- 172.21.15.66:0/2041761979 >> 172.21.15.66:0/2041761979 conn(0x7f12580617a0 msgr2=0x7f12580b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:23.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.446+0000 7f125fc25640 1 -- 172.21.15.66:0/2041761979 shutdown_connections 2024-03-25T18:14:23.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.446+0000 7f125fc25640 1 -- 172.21.15.66:0/2041761979 wait complete. 2024-03-25T18:14:23.447 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:23.446+0000 7f125fc25640 1 librados: shutdown 2024-03-25T18:14:23.458 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:30.264 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:30.382 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.381+0000 7fbe54400640 10 monclient: get_monmap_and_config 2024-03-25T18:14:30.382 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.381+0000 7fbe54400640 10 monclient: build_initial_monmap 2024-03-25T18:14:30.382 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.381+0000 7fbe54400640 10 monclient: monmap: 2024-03-25T18:14:30.382 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:14:30.382 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:30.382 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:14:30.381913+0000 2024-03-25T18:14:30.382 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:14:30.381913+0000 2024-03-25T18:14:30.382 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:30.382 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:30.382 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:14:30.382 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:14:30.382 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:14:30.383 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:30.384 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.382+0000 7fbe54400640 1 Processor -- start 2024-03-25T18:14:30.384 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.382+0000 7fbe54400640 1 -- start start 2024-03-25T18:14:30.384 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.382+0000 7fbe54400640 10 monclient: init 2024-03-25T18:14:30.384 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.382+0000 7fbe54400640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:30.384 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.382+0000 7fbe54400640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:30.384 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.382+0000 7fbe54400640 10 monclient(hunting): picked mon.noname-b con 0x7fbe4c0a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:14:30.384 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.383+0000 7fbe54400640 10 monclient(hunting): picked mon.noname-a con 0x7fbe4c13f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:14:30.384 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.383+0000 7fbe54400640 10 monclient(hunting): picked mon.noname-c con 0x7fbe4c1430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:14:30.384 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.383+0000 7fbe54400640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe4c142520 con 0x7fbe4c13f710 2024-03-25T18:14:30.384 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.383+0000 7fbe54400640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe4c0b28a0 con 0x7fbe4c1430c0 2024-03-25T18:14:30.384 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.383+0000 7fbe54400640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe4c0b2a30 con 0x7fbe4c0a98f0 2024-03-25T18:14:30.384 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.383+0000 7fbe54400640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:30.384 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.383+0000 7fbe54400640 10 monclient(hunting): authenticate will time out at 7619.442320s 2024-03-25T18:14:30.398 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.397+0000 7fbe5299d640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fbe4c1430c0 0x7fbe4c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:50200/0 (socket says 172.21.15.66:50200) 2024-03-25T18:14:30.398 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.397+0000 7fbe5299d640 1 -- 172.21.15.66:0/1052895535 learned_addr learned my addr 172.21.15.66:0/1052895535 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:14:30.414 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.414+0000 7fbe5119a640 1 -- 172.21.15.66:0/1052895535 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2978542315 0 0) 0x7fbe4c0b28a0 con 0x7fbe4c1430c0 2024-03-25T18:14:30.414 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.414+0000 7fbe5119a640 10 monclient(hunting): _negotiate 2024-03-25T18:14:30.414 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.414+0000 7fbe5119a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:30.414 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.414+0000 7fbe5119a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:30.414 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.414+0000 7fbe5119a640 10 monclient(hunting): my global_id is 4699 2024-03-25T18:14:30.414 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.414+0000 7fbe5119a640 1 -- 172.21.15.66:0/1052895535 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe38003520 con 0x7fbe4c1430c0 2024-03-25T18:14:30.416 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.415+0000 7fbe5119a640 1 -- 172.21.15.66:0/1052895535 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 970548114 0 0) 0x7fbe4c142520 con 0x7fbe4c13f710 2024-03-25T18:14:30.416 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.415+0000 7fbe5119a640 10 monclient(hunting): _negotiate 2024-03-25T18:14:30.416 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.415+0000 7fbe5119a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:30.416 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.415+0000 7fbe5119a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:30.416 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.415+0000 7fbe5119a640 10 monclient(hunting): my global_id is 4704 2024-03-25T18:14:30.416 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.415+0000 7fbe5119a640 1 -- 172.21.15.66:0/1052895535 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe4c0b28a0 con 0x7fbe4c13f710 2024-03-25T18:14:30.425 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.424+0000 7fbe5119a640 1 -- 172.21.15.66:0/1052895535 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2282293482 0 0) 0x7fbe4c0b2a30 con 0x7fbe4c0a98f0 2024-03-25T18:14:30.425 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.424+0000 7fbe5119a640 10 monclient(hunting): _negotiate 2024-03-25T18:14:30.425 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.424+0000 7fbe5119a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:30.425 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.424+0000 7fbe5119a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:30.425 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.424+0000 7fbe5119a640 10 monclient(hunting): my global_id is 4697 2024-03-25T18:14:30.426 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.424+0000 7fbe5119a640 1 -- 172.21.15.66:0/1052895535 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe4c142520 con 0x7fbe4c0a98f0 2024-03-25T18:14:30.442 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.441+0000 7fbe5119a640 1 -- 172.21.15.66:0/1052895535 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1242489107 0 0) 0x7fbe38003520 con 0x7fbe4c1430c0 2024-03-25T18:14:30.442 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.441+0000 7fbe5119a640 1 -- 172.21.15.66:0/1052895535 >> v1:172.21.15.66:6790/0 conn(0x7fbe4c0a98f0 legacy=0x7fbe4c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:30.442 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.441+0000 7fbe5119a640 1 -- 172.21.15.66:0/1052895535 >> v1:172.21.15.66:6789/0 conn(0x7fbe4c13f710 legacy=0x7fbe4c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:30.442 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.441+0000 7fbe5119a640 10 monclient: _finish_hunting 0 2024-03-25T18:14:30.442 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.441+0000 7fbe5119a640 1 monclient: found mon.noname-c 2024-03-25T18:14:30.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.441+0000 7fbe5119a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:30.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.441+0000 7fbe5119a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:14:30.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.441+0000 7fbe5119a640 1 -- 172.21.15.66:0/1052895535 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe4c0b44c0 con 0x7fbe4c1430c0 2024-03-25T18:14:30.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.441+0000 7fbe5119a640 10 monclient: _finish_auth 0 2024-03-25T18:14:30.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.441+0000 7fbe5119a640 10 monclient: _check_auth_tickets 2024-03-25T18:14:30.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.441+0000 7fbe5119a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:30.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.441+0000 7fbe5119a640 1 -- 172.21.15.66:0/1052895535 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fbe44001a10 con 0x7fbe4c1430c0 2024-03-25T18:14:30.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.441+0000 7fbe5119a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:30.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.441+0000 7fbe5119a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:30.443 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.442+0000 7fbe5119a640 10 monclient: dump: 2024-03-25T18:14:30.443 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:30.443 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:30.443 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:30.443 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:30.443 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:30.443 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:30.444 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:30.444 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:30.444 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:30.444 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:30.444 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.442+0000 7fbe54400640 5 monclient: authenticate success, global_id 4699 2024-03-25T18:14:30.444 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.442+0000 7fbe54400640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:30.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.467+0000 7fbe5119a640 1 -- 172.21.15.66:0/1052895535 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe44001b70 con 0x7fbe4c1430c0 2024-03-25T18:14:30.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.467+0000 7fbe5119a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:30.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.467+0000 7fbe5119a640 1 -- 172.21.15.66:0/1052895535 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fbe44001f80 con 0x7fbe4c1430c0 2024-03-25T18:14:30.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.467+0000 7fbe54400640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:30.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.467+0000 7fbe54400640 4 set_mon_vals no callback set 2024-03-25T18:14:30.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.467+0000 7fbe5119a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:30.467 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.467+0000 7fbe5119a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:30.468 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.467+0000 7fbe5119a640 10 monclient: dump: 2024-03-25T18:14:30.468 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:30.468 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:30.468 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:30.468 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:30.468 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:30.468 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:30.468 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:30.468 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:30.468 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:30.468 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:30.469 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.467+0000 7fbe54400640 10 monclient: shutdown 2024-03-25T18:14:30.469 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.467+0000 7fbe54400640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:30.469 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.467+0000 7fbe54400640 1 -- 172.21.15.66:0/1052895535 >> v1:172.21.15.193:6789/0 conn(0x7fbe4c1430c0 legacy=0x7fbe4c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:30.469 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.468+0000 7fbe54400640 1 -- 172.21.15.66:0/1052895535 shutdown_connections 2024-03-25T18:14:30.469 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.468+0000 7fbe54400640 1 -- 172.21.15.66:0/1052895535 >> 172.21.15.66:0/1052895535 conn(0x7fbe4c0b7ea0 msgr2=0x7fbe4c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:30.469 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.468+0000 7fbe54400640 1 -- 172.21.15.66:0/1052895535 shutdown_connections 2024-03-25T18:14:30.470 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.468+0000 7fbe54400640 1 -- 172.21.15.66:0/1052895535 wait complete. 2024-03-25T18:14:30.470 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.469+0000 7fbe54400640 10 monclient: build_initial_monmap 2024-03-25T18:14:30.470 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.469+0000 7fbe54400640 10 monclient: monmap: 2024-03-25T18:14:30.470 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:14:30.470 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:30.470 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:14:30.470 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:14:30.470 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:30.470 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:14:30.470 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:14:30.470 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:14:30.470 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:14:30.470 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:30.472 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.469+0000 7fbe54400640 1 librados: starting msgr at 2024-03-25T18:14:30.472 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.469+0000 7fbe54400640 1 librados: starting objecter 2024-03-25T18:14:30.472 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.469+0000 7fbe54400640 1 Processor -- start 2024-03-25T18:14:30.472 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.470+0000 7fbe54400640 1 -- start start 2024-03-25T18:14:30.472 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.470+0000 7fbe54400640 1 librados: setting wanted keys 2024-03-25T18:14:30.472 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.470+0000 7fbe54400640 1 librados: calling monclient init 2024-03-25T18:14:30.472 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.470+0000 7fbe54400640 10 monclient: init 2024-03-25T18:14:30.472 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.470+0000 7fbe54400640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:30.472 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.470+0000 7fbe54400640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:30.472 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.470+0000 7fbe54400640 10 monclient(hunting): picked mon.noname-a con 0x7fbe4c0a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:14:30.472 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.470+0000 7fbe54400640 10 monclient(hunting): picked mon.noname-b con 0x7fbe4c13f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:14:30.472 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.470+0000 7fbe54400640 10 monclient(hunting): picked mon.noname-c con 0x7fbe4c1430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:14:30.472 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.470+0000 7fbe54400640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe4c0abd30 con 0x7fbe4c0a98f0 2024-03-25T18:14:30.472 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.470+0000 7fbe54400640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe4c1d7810 con 0x7fbe4c13f710 2024-03-25T18:14:30.473 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.470+0000 7fbe54400640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe4c1d8440 con 0x7fbe4c1430c0 2024-03-25T18:14:30.473 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.470+0000 7fbe54400640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:30.473 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.470+0000 7fbe54400640 10 monclient(hunting): authenticate will time out at 7619.529941s 2024-03-25T18:14:30.473 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.473+0000 7fbe5219c640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7fbe4c0a98f0 0x7fbe4c1d7d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:45858/0 (socket says 172.21.15.66:45858) 2024-03-25T18:14:30.473 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.473+0000 7fbe5219c640 1 -- 172.21.15.66:0/3114229771 learned_addr learned my addr 172.21.15.66:0/3114229771 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:14:30.473 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.473+0000 7fbe42ffd640 10 client.?.objecter ms_handle_connect 0x7fbe4c0a98f0 2024-03-25T18:14:30.473 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.473+0000 7fbe42ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:30.481 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.480+0000 7fbe42ffd640 1 -- 172.21.15.66:0/3114229771 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715672180 0 0) 0x7fbe4c0abd30 con 0x7fbe4c0a98f0 2024-03-25T18:14:30.481 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.480+0000 7fbe42ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:30.481 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.480+0000 7fbe42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:30.481 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.480+0000 7fbe42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:30.481 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.480+0000 7fbe42ffd640 10 monclient(hunting): my global_id is 4707 2024-03-25T18:14:30.481 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.480+0000 7fbe42ffd640 1 -- 172.21.15.66:0/3114229771 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe30004eb0 con 0x7fbe4c0a98f0 2024-03-25T18:14:30.484 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.483+0000 7fbe42ffd640 10 client.?.objecter ms_handle_connect 0x7fbe4c13f710 2024-03-25T18:14:30.484 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.483+0000 7fbe42ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:30.489 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.488+0000 7fbe42ffd640 10 client.?.objecter ms_handle_connect 0x7fbe4c1430c0 2024-03-25T18:14:30.489 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.488+0000 7fbe42ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:30.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.495+0000 7fbe42ffd640 1 -- 172.21.15.66:0/3114229771 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2942101117 0 0) 0x7fbe30004eb0 con 0x7fbe4c0a98f0 2024-03-25T18:14:30.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.495+0000 7fbe42ffd640 1 -- 172.21.15.66:0/3114229771 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe4c0abd30 con 0x7fbe4c0a98f0 2024-03-25T18:14:30.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.495+0000 7fbe42ffd640 1 -- 172.21.15.66:0/3114229771 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fbe48001a10 con 0x7fbe4c0a98f0 2024-03-25T18:14:30.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.495+0000 7fbe42ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:30.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.495+0000 7fbe42ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:30.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.495+0000 7fbe42ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:30.496 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:30.497 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:30.497 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:30.497 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:30.497 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:30.497 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:30.497 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:30.497 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:30.497 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:30.497 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:30.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.495+0000 7fbe42ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:30.502 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.502+0000 7fbe42ffd640 1 -- 172.21.15.66:0/3114229771 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3790650172 0 0) 0x7fbe4c1d7810 con 0x7fbe4c13f710 2024-03-25T18:14:30.502 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.502+0000 7fbe42ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:30.502 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.502+0000 7fbe42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:30.502 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.502+0000 7fbe42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:30.503 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.502+0000 7fbe42ffd640 10 monclient(hunting): my global_id is 4702 2024-03-25T18:14:30.503 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.502+0000 7fbe42ffd640 1 -- 172.21.15.66:0/3114229771 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe30004eb0 con 0x7fbe4c13f710 2024-03-25T18:14:30.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.515+0000 7fbe42ffd640 1 -- 172.21.15.66:0/3114229771 <== mon.0 v1:172.21.15.66:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 788252868 0 0) 0x7fbe4c0abd30 con 0x7fbe4c0a98f0 2024-03-25T18:14:30.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.515+0000 7fbe42ffd640 1 -- 172.21.15.66:0/3114229771 >> v1:172.21.15.66:6790/0 conn(0x7fbe4c1430c0 legacy=0x7fbe4c1383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:30.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.515+0000 7fbe42ffd640 1 -- 172.21.15.66:0/3114229771 >> v1:172.21.15.193:6789/0 conn(0x7fbe4c13f710 legacy=0x7fbe4c1d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:30.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.515+0000 7fbe42ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:30.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.515+0000 7fbe42ffd640 1 monclient: found mon.a 2024-03-25T18:14:30.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.515+0000 7fbe42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:30.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.515+0000 7fbe42ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:14:30.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.515+0000 7fbe42ffd640 1 -- 172.21.15.66:0/3114229771 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe4c1d5000 con 0x7fbe4c0a98f0 2024-03-25T18:14:30.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.515+0000 7fbe42ffd640 10 monclient: _finish_auth 0 2024-03-25T18:14:30.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.515+0000 7fbe42ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:30.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.515+0000 7fbe42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:30.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.515+0000 7fbe54400640 5 monclient: authenticate success, global_id 4707 2024-03-25T18:14:30.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.515+0000 7fbe54400640 10 monclient: _renew_subs 2024-03-25T18:14:30.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.515+0000 7fbe54400640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:14:30.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.515+0000 7fbe54400640 1 -- 172.21.15.66:0/3114229771 --> v1:172.21.15.66:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe4c1d5250 con 0x7fbe4c0a98f0 2024-03-25T18:14:30.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.515+0000 7fbe54400640 10 client.4707.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:30.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.515+0000 7fbe54400640 10 monclient: _renew_subs 2024-03-25T18:14:30.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.515+0000 7fbe54400640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:14:30.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.515+0000 7fbe54400640 1 -- 172.21.15.66:0/3114229771 --> v1:172.21.15.66:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe4c1d43c0 con 0x7fbe4c0a98f0 2024-03-25T18:14:30.517 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.515+0000 7fbe54400640 1 librados: init done 2024-03-25T18:14:30.518 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.516+0000 7fbe54400640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:30.518 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.516+0000 7fbe54400640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:30.518 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.516+0000 7fbe54400640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:14:30.518 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.516+0000 7fbe54400640 1 -- 172.21.15.66:0/3114229771 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe4c1d7670 con 0x7fbe4c0a98f0 2024-03-25T18:14:30.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.866+0000 7fbe42ffd640 1 -- 172.21.15.66:0/3114229771 <== mon.0 v1:172.21.15.66:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe48001cb0 con 0x7fbe4c0a98f0 2024-03-25T18:14:30.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.866+0000 7fbe42ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:30.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.866+0000 7fbe42ffd640 1 -- 172.21.15.66:0/3114229771 <== mon.0 v1:172.21.15.66:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7fbe480038c0 con 0x7fbe4c0a98f0 2024-03-25T18:14:30.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.866+0000 7fbe42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:30.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.866+0000 7fbe42ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:30.867 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.866+0000 7fbe42ffd640 10 monclient: dump: 2024-03-25T18:14:30.867 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:30.868 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:30.868 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:30.868 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:30.868 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:30.868 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:30.868 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:30.868 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:30.868 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:30.868 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:30.868 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.866+0000 7fbe50999640 4 set_mon_vals no callback set 2024-03-25T18:14:30.869 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.867+0000 7fbe42ffd640 1 -- 172.21.15.66:0/3114229771 <== mon.0 v1:172.21.15.66:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7fbe48003ae0 con 0x7fbe4c0a98f0 2024-03-25T18:14:30.869 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.868+0000 7fbe5199b640 10 monclient: get_auth_request con 0x7fbe3006dd90 auth_method 0 2024-03-25T18:14:30.869 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.868+0000 7fbe42ffd640 10 client.4707.objecter ms_handle_connect 0x7fbe3006dd90 2024-03-25T18:14:30.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.873+0000 7fbe42ffd640 1 -- 172.21.15.66:0/3114229771 <== mon.0 v1:172.21.15.66:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4152745027 0 0) 0x7fbe480546d0 con 0x7fbe4c0a98f0 2024-03-25T18:14:30.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.873+0000 7fbe42ffd640 10 client.4707.objecter ms_dispatch 0x7fbe4c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:14:30.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.873+0000 7fbe42ffd640 3 client.4707.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:14:30.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.873+0000 7fbe42ffd640 3 client.4707.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:14:30.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.873+0000 7fbe42ffd640 20 client.4707.objecter dump_active .. 0 homeless 2024-03-25T18:14:30.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.873+0000 7fbe42ffd640 1 -- 172.21.15.66:0/3114229771 <== mon.0 v1:172.21.15.66:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbe480501e0 con 0x7fbe4c0a98f0 2024-03-25T18:14:30.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.873+0000 7fbe42ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:30.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:30.873+0000 7fbe42ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:31.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:31.089+0000 7fbe54400640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:31.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:31.089+0000 7fbe54400640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:31.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:31.089+0000 7fbe54400640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:14:31.089 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:31.089+0000 7fbe54400640 1 -- 172.21.15.66:0/3114229771 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe4c0abb70 con 0x7fbe4c0a98f0 2024-03-25T18:14:31.159 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:31.158+0000 7fbe42ffd640 1 -- 172.21.15.66:0/3114229771 <== mon.0 v1:172.21.15.66:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4258808387) 0x7fbe48052110 con 0x7fbe4c0a98f0 2024-03-25T18:14:31.159 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:31.158+0000 7fbe42ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:31.159 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:31.158+0000 7fbe42ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:31.160 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:14:31.160 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":11,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:13:50.353085+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:50.353085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:31.162 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:31.161+0000 7fbe54400640 10 librados: watch_flush enter 2024-03-25T18:14:31.162 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:31.161+0000 7fbe54400640 10 librados: watch_flush exit 2024-03-25T18:14:31.162 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:31.161+0000 7fbe54400640 20 client.4707.objecter shutdown clearing up homeless session... 2024-03-25T18:14:31.162 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:31.161+0000 7fbe54400640 10 client.4707.objecter successfully canceled tick 2024-03-25T18:14:31.162 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:31.161+0000 7fbe54400640 1 -- 172.21.15.66:0/3114229771 >> v1:172.21.15.193:6800/2275676803 conn(0x7fbe3006dd90 legacy=0x7fbe30070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:31.162 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:31.161+0000 7fbe54400640 10 monclient: shutdown 2024-03-25T18:14:31.162 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:31.161+0000 7fbe54400640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:31.162 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:31.161+0000 7fbe54400640 1 -- 172.21.15.66:0/3114229771 >> v1:172.21.15.66:6789/0 conn(0x7fbe4c0a98f0 legacy=0x7fbe4c1d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:31.162 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:31.161+0000 7fbe54400640 1 -- 172.21.15.66:0/3114229771 shutdown_connections 2024-03-25T18:14:31.162 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:31.161+0000 7fbe54400640 1 -- 172.21.15.66:0/3114229771 >> 172.21.15.66:0/3114229771 conn(0x7fbe4c0b7ea0 msgr2=0x7fbe4c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:31.163 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:31.161+0000 7fbe54400640 1 -- 172.21.15.66:0/3114229771 shutdown_connections 2024-03-25T18:14:31.163 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:31.161+0000 7fbe54400640 1 -- 172.21.15.66:0/3114229771 wait complete. 2024-03-25T18:14:31.163 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:31.161+0000 7fbe54400640 1 librados: shutdown 2024-03-25T18:14:31.173 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:35.607 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:11:06.587 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:35.613 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:11:06.593 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:35.645 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:11:06.604 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:35.656 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:11:06.636 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:35.657 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:11:06.616 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:35.683 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:11:06.642 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:35.688 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:11:06.668 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:35.697 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:11:06.678 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:35.725 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:11:06.705 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:37.980 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:38.103 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.102+0000 7f673b37b640 10 monclient: get_monmap_and_config 2024-03-25T18:14:38.103 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.102+0000 7f673b37b640 10 monclient: build_initial_monmap 2024-03-25T18:14:38.103 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.102+0000 7f673b37b640 10 monclient: monmap: 2024-03-25T18:14:38.103 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:14:38.103 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:38.103 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:14:38.103124+0000 2024-03-25T18:14:38.103 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:14:38.103124+0000 2024-03-25T18:14:38.103 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:38.104 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:38.104 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:14:38.104 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:14:38.104 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:14:38.104 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:38.105 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.103+0000 7f673b37b640 1 Processor -- start 2024-03-25T18:14:38.105 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.103+0000 7f673b37b640 1 -- start start 2024-03-25T18:14:38.105 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.103+0000 7f673b37b640 10 monclient: init 2024-03-25T18:14:38.106 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.104+0000 7f673b37b640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:38.106 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.104+0000 7f673b37b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:38.106 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.104+0000 7f673b37b640 10 monclient(hunting): picked mon.noname-a con 0x7f67340a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:14:38.106 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.104+0000 7f673b37b640 10 monclient(hunting): picked mon.noname-b con 0x7f673413f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:14:38.106 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.104+0000 7f673b37b640 10 monclient(hunting): picked mon.noname-c con 0x7f67341430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:14:38.106 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.104+0000 7f673b37b640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6734142520 con 0x7f67340a98f0 2024-03-25T18:14:38.106 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.104+0000 7f673b37b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67340b28a0 con 0x7f67341430c0 2024-03-25T18:14:38.106 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.104+0000 7f673b37b640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67340b2a30 con 0x7f673413f710 2024-03-25T18:14:38.106 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.104+0000 7f673b37b640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:38.106 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.104+0000 7f673b37b640 10 monclient(hunting): authenticate will time out at 7627.163463s 2024-03-25T18:14:38.117 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.116+0000 7f6738916640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f673413f710 0x7f67341429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:47218/0 (socket says 172.21.15.66:47218) 2024-03-25T18:14:38.117 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.116+0000 7f6738916640 1 -- 172.21.15.66:0/1600957295 learned_addr learned my addr 172.21.15.66:0/1600957295 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:14:38.124 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.124+0000 7f672bfff640 1 -- 172.21.15.66:0/1600957295 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1249213804 0 0) 0x7f67340b2a30 con 0x7f673413f710 2024-03-25T18:14:38.124 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.124+0000 7f672bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:38.125 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.124+0000 7f672bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:38.125 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.124+0000 7f672bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:38.125 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.124+0000 7f672bfff640 10 monclient(hunting): my global_id is 4703 2024-03-25T18:14:38.125 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.124+0000 7f672bfff640 1 -- 172.21.15.66:0/1600957295 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6720003520 con 0x7f673413f710 2024-03-25T18:14:38.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.137+0000 7f672bfff640 1 -- 172.21.15.66:0/1600957295 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 530007973 0 0) 0x7f6734142520 con 0x7f67340a98f0 2024-03-25T18:14:38.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.137+0000 7f672bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:38.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.137+0000 7f672bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:38.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.137+0000 7f672bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:38.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.137+0000 7f672bfff640 10 monclient(hunting): my global_id is 4710 2024-03-25T18:14:38.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.137+0000 7f672bfff640 1 -- 172.21.15.66:0/1600957295 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67340b2a30 con 0x7f67340a98f0 2024-03-25T18:14:38.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.137+0000 7f672bfff640 1 -- 172.21.15.66:0/1600957295 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2582858643 0 0) 0x7f67340b28a0 con 0x7f67341430c0 2024-03-25T18:14:38.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.137+0000 7f672bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:38.138 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.137+0000 7f672bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:38.139 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.137+0000 7f672bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:38.139 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.137+0000 7f672bfff640 10 monclient(hunting): my global_id is 4705 2024-03-25T18:14:38.139 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.137+0000 7f672bfff640 1 -- 172.21.15.66:0/1600957295 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6734142520 con 0x7f67341430c0 2024-03-25T18:14:38.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.143+0000 7f672bfff640 1 -- 172.21.15.66:0/1600957295 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 928349350 0 0) 0x7f6720003520 con 0x7f673413f710 2024-03-25T18:14:38.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.143+0000 7f672bfff640 1 -- 172.21.15.66:0/1600957295 >> v1:172.21.15.193:6789/0 conn(0x7f67341430c0 legacy=0x7f673414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:38.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.143+0000 7f672bfff640 1 -- 172.21.15.66:0/1600957295 >> v1:172.21.15.66:6789/0 conn(0x7f67340a98f0 legacy=0x7f67340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:38.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.143+0000 7f672bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:14:38.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.143+0000 7f672bfff640 1 monclient: found mon.noname-b 2024-03-25T18:14:38.144 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.143+0000 7f672bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:38.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.143+0000 7f672bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.66:6790/0 2024-03-25T18:14:38.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.143+0000 7f672bfff640 1 -- 172.21.15.66:0/1600957295 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67340b44c0 con 0x7f673413f710 2024-03-25T18:14:38.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.143+0000 7f672bfff640 10 monclient: _finish_auth 0 2024-03-25T18:14:38.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.143+0000 7f672bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:14:38.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.143+0000 7f672bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:38.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.143+0000 7f672bfff640 1 -- 172.21.15.66:0/1600957295 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f6724001690 con 0x7f673413f710 2024-03-25T18:14:38.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.143+0000 7f673b37b640 5 monclient: authenticate success, global_id 4703 2024-03-25T18:14:38.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.143+0000 7f673b37b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:38.145 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.143+0000 7f672bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:38.146 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.143+0000 7f672bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:38.146 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.143+0000 7f672bfff640 10 monclient: dump: 2024-03-25T18:14:38.146 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:38.146 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:38.146 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:38.146 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:38.146 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:38.146 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:38.146 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:38.147 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:38.147 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:38.147 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:38.147 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.143+0000 7f673b37b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:38.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.149+0000 7f672bfff640 1 -- 172.21.15.66:0/1600957295 <== mon.2 v1:172.21.15.66:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67240017f0 con 0x7f673413f710 2024-03-25T18:14:38.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.149+0000 7f672bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:38.150 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.149+0000 7f672bfff640 1 -- 172.21.15.66:0/1600957295 <== mon.2 v1:172.21.15.66:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f6724001c00 con 0x7f673413f710 2024-03-25T18:14:38.151 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.149+0000 7f672bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:38.151 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.149+0000 7f672bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:38.151 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.149+0000 7f672bfff640 10 monclient: dump: 2024-03-25T18:14:38.151 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:38.151 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:38.151 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:38.151 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:38.151 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:38.151 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:38.151 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:38.151 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:38.152 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:38.152 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:38.152 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.150+0000 7f673b37b640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:38.152 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.150+0000 7f673b37b640 4 set_mon_vals no callback set 2024-03-25T18:14:38.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.150+0000 7f673b37b640 10 monclient: shutdown 2024-03-25T18:14:38.154 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.150+0000 7f673b37b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:38.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.150+0000 7f673b37b640 1 -- 172.21.15.66:0/1600957295 >> v1:172.21.15.66:6790/0 conn(0x7f673413f710 legacy=0x7f67341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:38.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.150+0000 7f673b37b640 1 -- 172.21.15.66:0/1600957295 shutdown_connections 2024-03-25T18:14:38.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.150+0000 7f673b37b640 1 -- 172.21.15.66:0/1600957295 >> 172.21.15.66:0/1600957295 conn(0x7f67340b7ea0 msgr2=0x7f67340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:38.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.151+0000 7f673b37b640 1 -- 172.21.15.66:0/1600957295 shutdown_connections 2024-03-25T18:14:38.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.151+0000 7f673b37b640 1 -- 172.21.15.66:0/1600957295 wait complete. 2024-03-25T18:14:38.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.151+0000 7f673b37b640 10 monclient: build_initial_monmap 2024-03-25T18:14:38.155 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.151+0000 7f673b37b640 10 monclient: monmap: 2024-03-25T18:14:38.155 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:14:38.155 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:38.155 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:14:38.156 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:14:38.156 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:38.156 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:14:38.156 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:14:38.156 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:14:38.156 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:14:38.156 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:38.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.152+0000 7f673b37b640 1 librados: starting msgr at 2024-03-25T18:14:38.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.152+0000 7f673b37b640 1 librados: starting objecter 2024-03-25T18:14:38.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.152+0000 7f673b37b640 1 Processor -- start 2024-03-25T18:14:38.156 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.152+0000 7f673b37b640 1 -- start start 2024-03-25T18:14:38.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.152+0000 7f673b37b640 1 librados: setting wanted keys 2024-03-25T18:14:38.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.152+0000 7f673b37b640 1 librados: calling monclient init 2024-03-25T18:14:38.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.152+0000 7f673b37b640 10 monclient: init 2024-03-25T18:14:38.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.152+0000 7f673b37b640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:38.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.152+0000 7f673b37b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:38.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.153+0000 7f673b37b640 10 monclient(hunting): picked mon.noname-c con 0x7f67340a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:14:38.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.153+0000 7f673b37b640 10 monclient(hunting): picked mon.noname-a con 0x7f673413f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:14:38.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.153+0000 7f673b37b640 10 monclient(hunting): picked mon.noname-b con 0x7f67341430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:14:38.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.153+0000 7f673b37b640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67340b1870 con 0x7f673413f710 2024-03-25T18:14:38.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.153+0000 7f673b37b640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67340ac910 con 0x7f67341430c0 2024-03-25T18:14:38.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.153+0000 7f673b37b640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67340abb00 con 0x7f67340a98f0 2024-03-25T18:14:38.157 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.153+0000 7f673b37b640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:38.158 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.153+0000 7f673b37b640 10 monclient(hunting): authenticate will time out at 7627.212273s 2024-03-25T18:14:38.172 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.172+0000 7f6738916640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f673413f710 0x7f67340ac200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:46292/0 (socket says 172.21.15.66:46292) 2024-03-25T18:14:38.173 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.172+0000 7f6738916640 1 -- 172.21.15.66:0/2952540963 learned_addr learned my addr 172.21.15.66:0/2952540963 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:14:38.174 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.174+0000 7f6729ffb640 10 client.?.objecter ms_handle_connect 0x7f673413f710 2024-03-25T18:14:38.175 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.174+0000 7f6729ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:38.175 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.175+0000 7f6729ffb640 10 client.?.objecter ms_handle_connect 0x7f67340a98f0 2024-03-25T18:14:38.175 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.175+0000 7f6729ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:38.178 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.178+0000 7f6729ffb640 10 client.?.objecter ms_handle_connect 0x7f67341430c0 2024-03-25T18:14:38.178 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.178+0000 7f6729ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:38.190 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.188+0000 7f6729ffb640 1 -- 172.21.15.66:0/2952540963 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2311901785 0 0) 0x7f67340ac910 con 0x7f67341430c0 2024-03-25T18:14:38.190 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.189+0000 7f6729ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:38.190 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.189+0000 7f6729ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:38.190 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.189+0000 7f6729ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:38.190 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.189+0000 7f6729ffb640 10 monclient(hunting): my global_id is 4708 2024-03-25T18:14:38.190 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.189+0000 7f6729ffb640 1 -- 172.21.15.66:0/2952540963 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6718004eb0 con 0x7f67341430c0 2024-03-25T18:14:38.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.190+0000 7f6729ffb640 1 -- 172.21.15.66:0/2952540963 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 447254544 0 0) 0x7f67340b44c0 con 0x7f673413f710 2024-03-25T18:14:38.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.190+0000 7f6729ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:38.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.190+0000 7f6729ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:38.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.190+0000 7f6729ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:38.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.190+0000 7f6729ffb640 10 monclient(hunting): my global_id is 4713 2024-03-25T18:14:38.191 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.190+0000 7f6729ffb640 1 -- 172.21.15.66:0/2952540963 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67340ac910 con 0x7f673413f710 2024-03-25T18:14:38.195 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.194+0000 7f6729ffb640 1 -- 172.21.15.66:0/2952540963 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 621935087 0 0) 0x7f67340abb00 con 0x7f67340a98f0 2024-03-25T18:14:38.195 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.195+0000 7f6729ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:38.195 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.195+0000 7f6729ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:38.195 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.195+0000 7f6729ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:38.195 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.195+0000 7f6729ffb640 10 monclient(hunting): my global_id is 4706 2024-03-25T18:14:38.196 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.195+0000 7f6729ffb640 1 -- 172.21.15.66:0/2952540963 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67340b44c0 con 0x7f67340a98f0 2024-03-25T18:14:38.205 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.204+0000 7f6729ffb640 1 -- 172.21.15.66:0/2952540963 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3786608488 0 0) 0x7f6718004eb0 con 0x7f67341430c0 2024-03-25T18:14:38.205 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.204+0000 7f6729ffb640 1 -- 172.21.15.66:0/2952540963 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67340abb00 con 0x7f67341430c0 2024-03-25T18:14:38.205 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.204+0000 7f6729ffb640 1 -- 172.21.15.66:0/2952540963 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f672c001bd0 con 0x7f67341430c0 2024-03-25T18:14:38.205 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.204+0000 7f6729ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:38.205 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.204+0000 7f6729ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:38.205 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.204+0000 7f6729ffb640 10 monclient(hunting): dump: 2024-03-25T18:14:38.205 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:38.205 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:38.205 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:38.205 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:38.206 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:38.206 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:38.206 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:38.206 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:38.206 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:38.206 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:38.206 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.204+0000 7f6729ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:38.212 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.211+0000 7f6729ffb640 1 -- 172.21.15.66:0/2952540963 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 517600483 0 0) 0x7f67340b44c0 con 0x7f67340a98f0 2024-03-25T18:14:38.212 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.211+0000 7f6729ffb640 1 -- 172.21.15.66:0/2952540963 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6718004eb0 con 0x7f67340a98f0 2024-03-25T18:14:38.212 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.211+0000 7f6729ffb640 1 -- 172.21.15.66:0/2952540963 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f67300018f0 con 0x7f67340a98f0 2024-03-25T18:14:38.212 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.211+0000 7f6729ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:38.212 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.211+0000 7f6729ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:38.212 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.211+0000 7f6729ffb640 10 monclient(hunting): dump: 2024-03-25T18:14:38.212 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:38.212 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:38.212 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:38.213 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:38.213 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:38.213 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:38.213 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:38.213 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:38.213 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:38.213 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:38.217 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.216+0000 7f6729ffb640 1 -- 172.21.15.66:0/2952540963 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2981157664 0 0) 0x7f67340ac910 con 0x7f673413f710 2024-03-25T18:14:38.217 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.217+0000 7f6729ffb640 1 -- 172.21.15.66:0/2952540963 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67340b44c0 con 0x7f673413f710 2024-03-25T18:14:38.217 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.217+0000 7f6729ffb640 1 -- 172.21.15.66:0/2952540963 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f67240019a0 con 0x7f673413f710 2024-03-25T18:14:38.217 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.217+0000 7f6729ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:38.218 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.217+0000 7f6729ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:38.218 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.217+0000 7f6729ffb640 10 monclient(hunting): dump: 2024-03-25T18:14:38.218 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:38.218 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:38.218 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:38.218 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:38.218 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:38.218 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:38.218 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:38.218 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:38.219 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:38.219 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:38.221 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.217+0000 7f6729ffb640 1 -- 172.21.15.66:0/2952540963 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2570830089 0 0) 0x7f67340abb00 con 0x7f67341430c0 2024-03-25T18:14:38.221 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.217+0000 7f6729ffb640 1 -- 172.21.15.66:0/2952540963 >> v1:172.21.15.66:6790/0 conn(0x7f67340a98f0 legacy=0x7f6734138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:38.221 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.217+0000 7f6729ffb640 1 -- 172.21.15.66:0/2952540963 >> v1:172.21.15.66:6789/0 conn(0x7f673413f710 legacy=0x7f67340ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:38.221 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.217+0000 7f6729ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:14:38.221 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.217+0000 7f6729ffb640 1 monclient: found mon.b 2024-03-25T18:14:38.221 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.217+0000 7f6729ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:38.221 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.217+0000 7f6729ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:14:38.221 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.217+0000 7f6729ffb640 1 -- 172.21.15.66:0/2952540963 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67340abde0 con 0x7f67341430c0 2024-03-25T18:14:38.221 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.217+0000 7f6729ffb640 10 monclient: _finish_auth 0 2024-03-25T18:14:38.221 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.217+0000 7f6729ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:14:38.221 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.217+0000 7f6729ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:38.222 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.217+0000 7f673b37b640 5 monclient: authenticate success, global_id 4708 2024-03-25T18:14:38.222 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.217+0000 7f673b37b640 10 monclient: _renew_subs 2024-03-25T18:14:38.222 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.217+0000 7f673b37b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:14:38.222 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.217+0000 7f673b37b640 1 -- 172.21.15.66:0/2952540963 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6734138f40 con 0x7f67341430c0 2024-03-25T18:14:38.222 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.218+0000 7f673b37b640 10 client.4708.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:38.222 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.218+0000 7f673b37b640 10 monclient: _renew_subs 2024-03-25T18:14:38.222 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.218+0000 7f673b37b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:14:38.222 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.218+0000 7f673b37b640 1 -- 172.21.15.66:0/2952540963 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f67341363f0 con 0x7f67341430c0 2024-03-25T18:14:38.222 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.218+0000 7f673b37b640 1 librados: init done 2024-03-25T18:14:38.222 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.219+0000 7f673b37b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:38.223 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.219+0000 7f673b37b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:38.223 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.219+0000 7f673b37b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:14:38.223 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.219+0000 7f673b37b640 1 -- 172.21.15.66:0/2952540963 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67340627b0 con 0x7f67341430c0 2024-03-25T18:14:38.509 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.508+0000 7f6729ffb640 1 -- 172.21.15.66:0/2952540963 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f672c002060 con 0x7f67341430c0 2024-03-25T18:14:38.509 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.508+0000 7f6729ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:38.509 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.508+0000 7f6729ffb640 1 -- 172.21.15.66:0/2952540963 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f672c003b30 con 0x7f67341430c0 2024-03-25T18:14:38.510 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.508+0000 7f6729ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:38.510 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.508+0000 7f6729ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:38.510 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.508+0000 7f6729ffb640 10 monclient: dump: 2024-03-25T18:14:38.510 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:38.510 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:38.510 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:38.510 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:38.510 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:38.510 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:38.511 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:38.511 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:38.511 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:38.511 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:38.511 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.508+0000 7f672b7fe640 4 set_mon_vals no callback set 2024-03-25T18:14:38.512 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.509+0000 7f6729ffb640 1 -- 172.21.15.66:0/2952540963 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f672c019b80 con 0x7f67341430c0 2024-03-25T18:14:38.512 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.510+0000 7f6739117640 10 monclient: get_auth_request con 0x7f671806e0b0 auth_method 0 2024-03-25T18:14:38.512 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.510+0000 7f6729ffb640 10 client.4708.objecter ms_handle_connect 0x7f671806e0b0 2024-03-25T18:14:38.515 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.514+0000 7f6729ffb640 1 -- 172.21.15.66:0/2952540963 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4152745027 0 0) 0x7f672c0538f0 con 0x7f67341430c0 2024-03-25T18:14:38.515 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.514+0000 7f6729ffb640 10 client.4708.objecter ms_dispatch 0x7f6734000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:14:38.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.514+0000 7f6729ffb640 3 client.4708.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:14:38.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.514+0000 7f6729ffb640 3 client.4708.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:14:38.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.514+0000 7f6729ffb640 20 client.4708.objecter dump_active .. 0 homeless 2024-03-25T18:14:38.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.515+0000 7f6729ffb640 1 -- 172.21.15.66:0/2952540963 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f672c0b08b0 con 0x7f67341430c0 2024-03-25T18:14:38.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.515+0000 7f6729ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:38.516 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.515+0000 7f6729ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:38.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.761+0000 7f673b37b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:38.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.761+0000 7f673b37b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:38.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.761+0000 7f673b37b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:14:38.762 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.761+0000 7f673b37b640 1 -- 172.21.15.66:0/2952540963 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67341434a0 con 0x7f67341430c0 2024-03-25T18:14:38.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.873+0000 7f6729ffb640 1 -- 172.21.15.66:0/2952540963 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4258808387) 0x7f672c052fc0 con 0x7f67341430c0 2024-03-25T18:14:38.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.873+0000 7f6729ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:38.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.873+0000 7f6729ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:38.874 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:14:38.874 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":11,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:13:50.353085+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:50.353085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:38.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.875+0000 7f673b37b640 10 librados: watch_flush enter 2024-03-25T18:14:38.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.875+0000 7f673b37b640 10 librados: watch_flush exit 2024-03-25T18:14:38.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.875+0000 7f673b37b640 20 client.4708.objecter shutdown clearing up homeless session... 2024-03-25T18:14:38.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.875+0000 7f673b37b640 10 client.4708.objecter successfully canceled tick 2024-03-25T18:14:38.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.875+0000 7f673b37b640 1 -- 172.21.15.66:0/2952540963 >> v1:172.21.15.193:6800/2275676803 conn(0x7f671806e0b0 legacy=0x7f6718070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:38.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.876+0000 7f673b37b640 10 monclient: shutdown 2024-03-25T18:14:38.878 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.876+0000 7f673b37b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:38.878 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.876+0000 7f673b37b640 1 -- 172.21.15.66:0/2952540963 >> v1:172.21.15.193:6789/0 conn(0x7f67341430c0 legacy=0x7f67341e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:38.878 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.876+0000 7f673b37b640 1 -- 172.21.15.66:0/2952540963 shutdown_connections 2024-03-25T18:14:38.878 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.876+0000 7f673b37b640 1 -- 172.21.15.66:0/2952540963 >> 172.21.15.66:0/2952540963 conn(0x7f67340b7ea0 msgr2=0x7f6734074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:38.878 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.876+0000 7f673b37b640 1 -- 172.21.15.66:0/2952540963 shutdown_connections 2024-03-25T18:14:38.878 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.876+0000 7f673b37b640 1 -- 172.21.15.66:0/2952540963 wait complete. 2024-03-25T18:14:38.878 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:38.876+0000 7f673b37b640 1 librados: shutdown 2024-03-25T18:14:38.888 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:45.694 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:45.812 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.811+0000 7f90da910640 10 monclient: get_monmap_and_config 2024-03-25T18:14:45.812 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.811+0000 7f90da910640 10 monclient: build_initial_monmap 2024-03-25T18:14:45.812 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.811+0000 7f90da910640 10 monclient: monmap: 2024-03-25T18:14:45.812 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:14:45.812 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:45.812 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:14:45.812185+0000 2024-03-25T18:14:45.812 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:14:45.812185+0000 2024-03-25T18:14:45.812 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:45.812 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:45.812 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:14:45.812 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:14:45.813 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:14:45.813 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:45.813 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.812+0000 7f90da910640 1 Processor -- start 2024-03-25T18:14:45.813 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.812+0000 7f90da910640 1 -- start start 2024-03-25T18:14:45.813 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.812+0000 7f90da910640 10 monclient: init 2024-03-25T18:14:45.814 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.813+0000 7f90da910640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:45.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.813+0000 7f90da910640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:45.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.813+0000 7f90da910640 10 monclient(hunting): picked mon.noname-b con 0x7f90d40a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:14:45.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.813+0000 7f90da910640 10 monclient(hunting): picked mon.noname-c con 0x7f90d413f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:14:45.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.813+0000 7f90da910640 10 monclient(hunting): picked mon.noname-a con 0x7f90d41430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:14:45.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.813+0000 7f90da910640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d4142520 con 0x7f90d41430c0 2024-03-25T18:14:45.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.813+0000 7f90da910640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d40b28a0 con 0x7f90d413f710 2024-03-25T18:14:45.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.813+0000 7f90da910640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d40b2a30 con 0x7f90d40a98f0 2024-03-25T18:14:45.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.813+0000 7f90da910640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:45.815 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.813+0000 7f90da910640 10 monclient(hunting): authenticate will time out at 7634.872949s 2024-03-25T18:14:45.821 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.821+0000 7f90d37fe640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f90d413f710 0x7f90d41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:45214/0 (socket says 172.21.15.66:45214) 2024-03-25T18:14:45.821 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.821+0000 7f90d37fe640 1 -- 172.21.15.66:0/345842843 learned_addr learned my addr 172.21.15.66:0/345842843 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:14:45.835 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.835+0000 7f90d2ffd640 1 -- 172.21.15.66:0/345842843 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3475681973 0 0) 0x7f90d40b28a0 con 0x7f90d413f710 2024-03-25T18:14:45.835 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.835+0000 7f90d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:45.835 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.835+0000 7f90d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:45.835 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.835+0000 7f90d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:45.836 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.835+0000 7f90d2ffd640 10 monclient(hunting): my global_id is 4711 2024-03-25T18:14:45.836 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.835+0000 7f90d2ffd640 1 -- 172.21.15.66:0/345842843 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90bc003520 con 0x7f90d413f710 2024-03-25T18:14:45.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.844+0000 7f90d2ffd640 1 -- 172.21.15.66:0/345842843 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3772780406 0 0) 0x7f90d40b2a30 con 0x7f90d40a98f0 2024-03-25T18:14:45.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.844+0000 7f90d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:45.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.844+0000 7f90d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:45.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.844+0000 7f90d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:45.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.844+0000 7f90d2ffd640 10 monclient(hunting): my global_id is 4709 2024-03-25T18:14:45.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.844+0000 7f90d2ffd640 1 -- 172.21.15.66:0/345842843 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90d40b28a0 con 0x7f90d40a98f0 2024-03-25T18:14:45.847 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.846+0000 7f90d2ffd640 1 -- 172.21.15.66:0/345842843 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3944205014 0 0) 0x7f90bc003520 con 0x7f90d413f710 2024-03-25T18:14:45.847 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.846+0000 7f90d2ffd640 1 -- 172.21.15.66:0/345842843 >> v1:172.21.15.66:6790/0 conn(0x7f90d40a98f0 legacy=0x7f90d40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:45.848 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.846+0000 7f90d2ffd640 1 -- 172.21.15.66:0/345842843 >> v1:172.21.15.66:6789/0 conn(0x7f90d41430c0 legacy=0x7f90d414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:45.848 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.846+0000 7f90d2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:45.848 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.846+0000 7f90d2ffd640 1 monclient: found mon.noname-c 2024-03-25T18:14:45.848 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.846+0000 7f90d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:45.848 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.846+0000 7f90d2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:14:45.848 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.846+0000 7f90d2ffd640 1 -- 172.21.15.66:0/345842843 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d40b44c0 con 0x7f90d413f710 2024-03-25T18:14:45.848 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.846+0000 7f90d2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:14:45.848 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.846+0000 7f90d2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:45.848 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.846+0000 7f90d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:45.848 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.846+0000 7f90d2ffd640 1 -- 172.21.15.66:0/345842843 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f90c8001690 con 0x7f90d413f710 2024-03-25T18:14:45.848 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.846+0000 7f90d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:45.848 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.846+0000 7f90d2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:45.848 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.846+0000 7f90d2ffd640 10 monclient: dump: 2024-03-25T18:14:45.848 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:45.848 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:45.848 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:45.849 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:45.849 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:45.849 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:45.849 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:45.849 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:45.849 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:45.849 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:45.849 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.846+0000 7f90da910640 5 monclient: authenticate success, global_id 4711 2024-03-25T18:14:45.849 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.847+0000 7f90da910640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:45.858 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.858+0000 7f90d2ffd640 1 -- 172.21.15.66:0/345842843 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90c80017f0 con 0x7f90d413f710 2024-03-25T18:14:45.858 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.858+0000 7f90d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:45.858 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.858+0000 7f90d2ffd640 1 -- 172.21.15.66:0/345842843 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f90c8001c00 con 0x7f90d413f710 2024-03-25T18:14:45.858 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.858+0000 7f90d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:45.859 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.858+0000 7f90d2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:45.859 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.858+0000 7f90d2ffd640 10 monclient: dump: 2024-03-25T18:14:45.859 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:45.859 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:45.859 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:45.859 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:45.859 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:45.859 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:45.859 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:45.859 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:45.859 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:45.859 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:45.859 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.858+0000 7f90da910640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:45.859 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.858+0000 7f90da910640 4 set_mon_vals no callback set 2024-03-25T18:14:45.861 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.858+0000 7f90da910640 10 monclient: shutdown 2024-03-25T18:14:45.861 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.858+0000 7f90da910640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:45.862 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.858+0000 7f90da910640 1 -- 172.21.15.66:0/345842843 >> v1:172.21.15.193:6789/0 conn(0x7f90d413f710 legacy=0x7f90d41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:45.862 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.858+0000 7f90da910640 1 -- 172.21.15.66:0/345842843 shutdown_connections 2024-03-25T18:14:45.862 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.858+0000 7f90da910640 1 -- 172.21.15.66:0/345842843 >> 172.21.15.66:0/345842843 conn(0x7f90d40b7ea0 msgr2=0x7f90d40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:45.862 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.859+0000 7f90da910640 1 -- 172.21.15.66:0/345842843 shutdown_connections 2024-03-25T18:14:45.862 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.859+0000 7f90da910640 1 -- 172.21.15.66:0/345842843 wait complete. 2024-03-25T18:14:45.862 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.859+0000 7f90da910640 10 monclient: build_initial_monmap 2024-03-25T18:14:45.862 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.859+0000 7f90da910640 10 monclient: monmap: 2024-03-25T18:14:45.862 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:14:45.862 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:45.862 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:14:45.862 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:14:45.862 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:45.862 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:14:45.863 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:14:45.863 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:14:45.863 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:14:45.863 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:45.863 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.859+0000 7f90da910640 1 librados: starting msgr at 2024-03-25T18:14:45.863 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.859+0000 7f90da910640 1 librados: starting objecter 2024-03-25T18:14:45.863 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.860+0000 7f90da910640 1 Processor -- start 2024-03-25T18:14:45.863 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.860+0000 7f90da910640 1 -- start start 2024-03-25T18:14:45.863 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.860+0000 7f90da910640 1 librados: setting wanted keys 2024-03-25T18:14:45.863 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.860+0000 7f90da910640 1 librados: calling monclient init 2024-03-25T18:14:45.863 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.860+0000 7f90da910640 10 monclient: init 2024-03-25T18:14:45.863 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.860+0000 7f90da910640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:45.863 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.860+0000 7f90da910640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:45.863 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.860+0000 7f90da910640 10 monclient(hunting): picked mon.noname-a con 0x7f90d40a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:14:45.863 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.860+0000 7f90da910640 10 monclient(hunting): picked mon.noname-b con 0x7f90d413f710 addr v1:172.21.15.193:6789/0 2024-03-25T18:14:45.863 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.860+0000 7f90da910640 10 monclient(hunting): picked mon.noname-c con 0x7f90d41430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:14:45.864 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.860+0000 7f90da910640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d40abde0 con 0x7f90d40a98f0 2024-03-25T18:14:45.864 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.860+0000 7f90da910640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d41d7730 con 0x7f90d413f710 2024-03-25T18:14:45.864 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.860+0000 7f90da910640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d41d8360 con 0x7f90d41430c0 2024-03-25T18:14:45.864 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.861+0000 7f90da910640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:45.864 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.861+0000 7f90da910640 10 monclient(hunting): authenticate will time out at 7634.920103s 2024-03-25T18:14:45.871 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.871+0000 7f90d3fff640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f90d40a98f0 0x7f90d41d7c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:59688/0 (socket says 172.21.15.66:59688) 2024-03-25T18:14:45.871 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.871+0000 7f90d3fff640 1 -- 172.21.15.66:0/76954401 learned_addr learned my addr 172.21.15.66:0/76954401 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:14:45.873 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.873+0000 7f90d0ff9640 10 client.?.objecter ms_handle_connect 0x7f90d40a98f0 2024-03-25T18:14:45.873 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.873+0000 7f90d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:45.876 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.876+0000 7f90d0ff9640 10 client.?.objecter ms_handle_connect 0x7f90d413f710 2024-03-25T18:14:45.877 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.876+0000 7f90d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:45.885 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.885+0000 7f90d0ff9640 1 -- 172.21.15.66:0/76954401 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4094801275 0 0) 0x7f90d40abde0 con 0x7f90d40a98f0 2024-03-25T18:14:45.885 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.885+0000 7f90d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:45.885 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.885+0000 7f90d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:45.886 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.885+0000 7f90d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:45.886 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.885+0000 7f90d0ff9640 10 monclient(hunting): my global_id is 4719 2024-03-25T18:14:45.886 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.885+0000 7f90d0ff9640 1 -- 172.21.15.66:0/76954401 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90c0004eb0 con 0x7f90d40a98f0 2024-03-25T18:14:45.886 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.885+0000 7f90d0ff9640 10 client.?.objecter ms_handle_connect 0x7f90d41430c0 2024-03-25T18:14:45.886 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.885+0000 7f90d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:45.890 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.889+0000 7f90d0ff9640 1 -- 172.21.15.66:0/76954401 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3602193740 0 0) 0x7f90d41d7730 con 0x7f90d413f710 2024-03-25T18:14:45.890 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.889+0000 7f90d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:45.890 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.889+0000 7f90d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:45.890 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.889+0000 7f90d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:45.890 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.889+0000 7f90d0ff9640 10 monclient(hunting): my global_id is 4714 2024-03-25T18:14:45.890 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.889+0000 7f90d0ff9640 1 -- 172.21.15.66:0/76954401 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90d40abde0 con 0x7f90d413f710 2024-03-25T18:14:45.911 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.911+0000 7f90d0ff9640 1 -- 172.21.15.66:0/76954401 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464839600 0 0) 0x7f90d41d8360 con 0x7f90d41430c0 2024-03-25T18:14:45.912 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.911+0000 7f90d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:45.912 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.911+0000 7f90d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:45.912 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.911+0000 7f90d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:45.912 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.911+0000 7f90d0ff9640 10 monclient(hunting): my global_id is 4712 2024-03-25T18:14:45.912 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.911+0000 7f90d0ff9640 1 -- 172.21.15.66:0/76954401 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90d41d7730 con 0x7f90d41430c0 2024-03-25T18:14:45.914 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.914+0000 7f90d0ff9640 1 -- 172.21.15.66:0/76954401 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3890641003 0 0) 0x7f90c0004eb0 con 0x7f90d40a98f0 2024-03-25T18:14:45.915 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.914+0000 7f90d0ff9640 1 -- 172.21.15.66:0/76954401 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90d41d8360 con 0x7f90d40a98f0 2024-03-25T18:14:45.915 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.914+0000 7f90d0ff9640 1 -- 172.21.15.66:0/76954401 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f90c4001a10 con 0x7f90d40a98f0 2024-03-25T18:14:45.915 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.914+0000 7f90d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:45.915 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.914+0000 7f90d0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:45.915 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.914+0000 7f90d0ff9640 10 monclient(hunting): dump: 2024-03-25T18:14:45.915 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:45.915 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:45.915 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:45.915 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:45.915 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:45.915 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:45.915 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:45.915 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:45.916 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:45.916 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:45.916 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.914+0000 7f90d0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:45.919 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.918+0000 7f90d0ff9640 1 -- 172.21.15.66:0/76954401 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2041120507 0 0) 0x7f90d40abde0 con 0x7f90d413f710 2024-03-25T18:14:45.919 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.918+0000 7f90d0ff9640 1 -- 172.21.15.66:0/76954401 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90c0004eb0 con 0x7f90d413f710 2024-03-25T18:14:45.919 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.918+0000 7f90d0ff9640 1 -- 172.21.15.66:0/76954401 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f90c8001b00 con 0x7f90d413f710 2024-03-25T18:14:45.919 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.918+0000 7f90d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:45.919 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.918+0000 7f90d0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:45.919 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.918+0000 7f90d0ff9640 10 monclient(hunting): dump: 2024-03-25T18:14:45.919 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:45.919 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:45.920 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:45.920 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:45.920 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:45.920 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:45.920 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:45.920 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:45.920 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:45.920 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:45.931 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.930+0000 7f90d0ff9640 1 -- 172.21.15.66:0/76954401 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2365164452 0 0) 0x7f90d41d7730 con 0x7f90d41430c0 2024-03-25T18:14:45.931 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.930+0000 7f90d0ff9640 1 -- 172.21.15.66:0/76954401 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90d40abde0 con 0x7f90d41430c0 2024-03-25T18:14:45.931 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.930+0000 7f90d0ff9640 1 -- 172.21.15.66:0/76954401 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f90cc001a90 con 0x7f90d41430c0 2024-03-25T18:14:45.931 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.930+0000 7f90d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:45.931 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.930+0000 7f90d0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:45.931 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.930+0000 7f90d0ff9640 10 monclient(hunting): dump: 2024-03-25T18:14:45.931 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:45.931 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:45.931 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:45.932 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:45.932 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:45.932 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:45.932 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:45.932 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:45.932 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:45.932 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:45.938 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.936+0000 7f90d0ff9640 1 -- 172.21.15.66:0/76954401 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 801858142 0 0) 0x7f90c0004eb0 con 0x7f90d413f710 2024-03-25T18:14:45.938 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.936+0000 7f90d0ff9640 1 -- 172.21.15.66:0/76954401 >> v1:172.21.15.66:6790/0 conn(0x7f90d41430c0 legacy=0x7f90d4138630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:45.938 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.936+0000 7f90d0ff9640 1 -- 172.21.15.66:0/76954401 >> v1:172.21.15.66:6789/0 conn(0x7f90d40a98f0 legacy=0x7f90d41d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:45.938 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.936+0000 7f90d0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:14:45.938 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.936+0000 7f90d0ff9640 1 monclient: found mon.b 2024-03-25T18:14:45.938 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.936+0000 7f90d0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:45.938 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.936+0000 7f90d0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:14:45.938 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.936+0000 7f90d0ff9640 1 -- 172.21.15.66:0/76954401 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d41d5000 con 0x7f90d413f710 2024-03-25T18:14:45.938 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.937+0000 7f90d0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:14:45.938 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.937+0000 7f90d0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:14:45.938 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.937+0000 7f90d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:45.939 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.937+0000 7f90da910640 5 monclient: authenticate success, global_id 4714 2024-03-25T18:14:45.939 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.937+0000 7f90da910640 10 monclient: _renew_subs 2024-03-25T18:14:45.939 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.937+0000 7f90da910640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:14:45.939 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.937+0000 7f90da910640 1 -- 172.21.15.66:0/76954401 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d4138d40 con 0x7f90d413f710 2024-03-25T18:14:45.939 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.937+0000 7f90da910640 10 client.4714.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:45.939 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.937+0000 7f90da910640 10 monclient: _renew_subs 2024-03-25T18:14:45.939 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.937+0000 7f90da910640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:14:45.939 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.937+0000 7f90da910640 1 -- 172.21.15.66:0/76954401 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d41d43c0 con 0x7f90d413f710 2024-03-25T18:14:45.939 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.937+0000 7f90da910640 1 librados: init done 2024-03-25T18:14:45.940 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.938+0000 7f90da910640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:45.940 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.938+0000 7f90da910640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:45.940 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.938+0000 7f90da910640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:14:45.940 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:45.938+0000 7f90da910640 1 -- 172.21.15.66:0/76954401 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90d41d7590 con 0x7f90d413f710 2024-03-25T18:14:46.207 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.207+0000 7f90d0ff9640 1 -- 172.21.15.66:0/76954401 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90c8004a90 con 0x7f90d413f710 2024-03-25T18:14:46.207 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.207+0000 7f90d0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:46.207 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.207+0000 7f90d0ff9640 1 -- 172.21.15.66:0/76954401 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f90c8001c80 con 0x7f90d413f710 2024-03-25T18:14:46.208 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.207+0000 7f90d0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:46.208 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.207+0000 7f90d0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:46.208 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.207+0000 7f90d0ff9640 10 monclient: dump: 2024-03-25T18:14:46.208 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:46.208 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:46.208 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:46.208 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:46.208 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:46.208 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:46.208 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:46.208 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:46.208 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:46.208 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:46.208 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.207+0000 7f90d27fc640 4 set_mon_vals no callback set 2024-03-25T18:14:46.209 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.207+0000 7f90d0ff9640 1 -- 172.21.15.66:0/76954401 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f90c801aa70 con 0x7f90d413f710 2024-03-25T18:14:46.209 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.208+0000 7f90d3fff640 10 monclient: get_auth_request con 0x7f90c006e110 auth_method 0 2024-03-25T18:14:46.209 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.208+0000 7f90d0ff9640 10 client.4714.objecter ms_handle_connect 0x7f90c006e110 2024-03-25T18:14:46.215 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.214+0000 7f90d0ff9640 1 -- 172.21.15.66:0/76954401 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4152745027 0 0) 0x7f90c8054630 con 0x7f90d413f710 2024-03-25T18:14:46.215 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.214+0000 7f90d0ff9640 10 client.4714.objecter ms_dispatch 0x7f90d4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:14:46.215 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.214+0000 7f90d0ff9640 3 client.4714.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:14:46.215 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.214+0000 7f90d0ff9640 3 client.4714.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:14:46.215 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.214+0000 7f90d0ff9640 20 client.4714.objecter dump_active .. 0 homeless 2024-03-25T18:14:46.215 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.214+0000 7f90d0ff9640 1 -- 172.21.15.66:0/76954401 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f90c804ff70 con 0x7f90d413f710 2024-03-25T18:14:46.215 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.214+0000 7f90d0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:46.215 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.214+0000 7f90d0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:46.471 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.470+0000 7f90da910640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:46.471 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.470+0000 7f90da910640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:46.471 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.470+0000 7f90da910640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:14:46.471 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.470+0000 7f90da910640 1 -- 172.21.15.66:0/76954401 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90d41d78c0 con 0x7f90d413f710 2024-03-25T18:14:46.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.497+0000 7f90d0ff9640 1 -- 172.21.15.66:0/76954401 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4258808387) 0x7f90c8052070 con 0x7f90d413f710 2024-03-25T18:14:46.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.497+0000 7f90d0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:46.498 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.497+0000 7f90d0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:46.498 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:14:46.498 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":11,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:13:50.353085+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:50.353085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:46.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.499+0000 7f90da910640 10 librados: watch_flush enter 2024-03-25T18:14:46.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.499+0000 7f90da910640 10 librados: watch_flush exit 2024-03-25T18:14:46.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.499+0000 7f90da910640 20 client.4714.objecter shutdown clearing up homeless session... 2024-03-25T18:14:46.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.499+0000 7f90da910640 10 client.4714.objecter successfully canceled tick 2024-03-25T18:14:46.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.499+0000 7f90da910640 1 -- 172.21.15.66:0/76954401 >> v1:172.21.15.193:6800/2275676803 conn(0x7f90c006e110 legacy=0x7f90c00705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:46.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.499+0000 7f90da910640 10 monclient: shutdown 2024-03-25T18:14:46.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.499+0000 7f90da910640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:46.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.499+0000 7f90da910640 1 -- 172.21.15.66:0/76954401 >> v1:172.21.15.193:6789/0 conn(0x7f90d413f710 legacy=0x7f90d41d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:46.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.499+0000 7f90da910640 1 -- 172.21.15.66:0/76954401 shutdown_connections 2024-03-25T18:14:46.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.499+0000 7f90da910640 1 -- 172.21.15.66:0/76954401 >> 172.21.15.66:0/76954401 conn(0x7f90d40b7ea0 msgr2=0x7f90d4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:46.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.500+0000 7f90da910640 1 -- 172.21.15.66:0/76954401 shutdown_connections 2024-03-25T18:14:46.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.500+0000 7f90da910640 1 -- 172.21.15.66:0/76954401 wait complete. 2024-03-25T18:14:46.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:46.500+0000 7f90da910640 1 librados: shutdown 2024-03-25T18:14:46.511 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:52.685 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:11:23.665 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:52.693 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:11:23.673 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:52.698 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:11:23.679 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:52.702 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:11:23.682 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:52.703 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:11:23.661 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:52.706 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:11:23.664 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:52.714 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:11:23.695 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:52.722 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:11:23.703 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:52.725 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:11:23.683 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:53.318 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:53.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.435+0000 7f4ed5940640 10 monclient: get_monmap_and_config 2024-03-25T18:14:53.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.435+0000 7f4ed5940640 10 monclient: build_initial_monmap 2024-03-25T18:14:53.437 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.435+0000 7f4ed5940640 10 monclient: monmap: 2024-03-25T18:14:53.437 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:14:53.437 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:53.437 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:14:53.437217+0000 2024-03-25T18:14:53.437 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:14:53.437217+0000 2024-03-25T18:14:53.437 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:53.437 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:53.437 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:14:53.437 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:14:53.438 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:14:53.438 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:53.438 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.436+0000 7f4ed5940640 1 Processor -- start 2024-03-25T18:14:53.438 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.436+0000 7f4ed5940640 1 -- start start 2024-03-25T18:14:53.438 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.436+0000 7f4ed5940640 10 monclient: init 2024-03-25T18:14:53.439 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.437+0000 7f4ed5940640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:53.439 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.437+0000 7f4ed5940640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:53.439 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.437+0000 7f4ed5940640 10 monclient(hunting): picked mon.noname-c con 0x7f4ed00a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:14:53.439 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.437+0000 7f4ed5940640 10 monclient(hunting): picked mon.noname-a con 0x7f4ed013f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:14:53.440 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.437+0000 7f4ed5940640 10 monclient(hunting): picked mon.noname-b con 0x7f4ed01430c0 addr v1:172.21.15.66:6790/0 2024-03-25T18:14:53.440 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.437+0000 7f4ed5940640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed0142520 con 0x7f4ed013f710 2024-03-25T18:14:53.440 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.437+0000 7f4ed5940640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed00b28a0 con 0x7f4ed00a98f0 2024-03-25T18:14:53.440 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.437+0000 7f4ed5940640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed00b2a30 con 0x7f4ed01430c0 2024-03-25T18:14:53.440 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.437+0000 7f4ed5940640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:53.440 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.437+0000 7f4ed5940640 10 monclient(hunting): authenticate will time out at 7642.497760s 2024-03-25T18:14:53.449 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.448+0000 7f4eceffd640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f4ed00a98f0 0x7f4ed00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:49594/0 (socket says 172.21.15.66:49594) 2024-03-25T18:14:53.449 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.448+0000 7f4eceffd640 1 -- 172.21.15.66:0/2711557688 learned_addr learned my addr 172.21.15.66:0/2711557688 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:14:53.462 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.461+0000 7f4ecdffb640 1 -- 172.21.15.66:0/2711557688 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2433941879 0 0) 0x7f4ed0142520 con 0x7f4ed013f710 2024-03-25T18:14:53.462 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.461+0000 7f4ecdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:53.462 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.461+0000 7f4ecdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:53.462 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.461+0000 7f4ecdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:53.463 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.461+0000 7f4ecdffb640 10 monclient(hunting): my global_id is 4722 2024-03-25T18:14:53.463 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.461+0000 7f4ecdffb640 1 -- 172.21.15.66:0/2711557688 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4eb8003520 con 0x7f4ed013f710 2024-03-25T18:14:53.472 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.470+0000 7f4ecdffb640 1 -- 172.21.15.66:0/2711557688 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2358547326 0 0) 0x7f4ed00b28a0 con 0x7f4ed00a98f0 2024-03-25T18:14:53.472 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.470+0000 7f4ecdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:53.472 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.470+0000 7f4ecdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:53.472 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.470+0000 7f4ecdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:53.472 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.470+0000 7f4ecdffb640 10 monclient(hunting): my global_id is 4717 2024-03-25T18:14:53.472 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.470+0000 7f4ecdffb640 1 -- 172.21.15.66:0/2711557688 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ed0142520 con 0x7f4ed00a98f0 2024-03-25T18:14:53.484 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.482+0000 7f4ecdffb640 1 -- 172.21.15.66:0/2711557688 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 467272604 0 0) 0x7f4ed00b2a30 con 0x7f4ed01430c0 2024-03-25T18:14:53.484 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.482+0000 7f4ecdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:53.484 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.482+0000 7f4ecdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:53.484 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.482+0000 7f4ecdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:53.484 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.482+0000 7f4ecdffb640 10 monclient(hunting): my global_id is 4715 2024-03-25T18:14:53.484 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.482+0000 7f4ecdffb640 1 -- 172.21.15.66:0/2711557688 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ed00b28a0 con 0x7f4ed01430c0 2024-03-25T18:14:53.485 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.483+0000 7f4ecdffb640 1 -- 172.21.15.66:0/2711557688 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1714931941 0 0) 0x7f4ed0142520 con 0x7f4ed00a98f0 2024-03-25T18:14:53.485 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.483+0000 7f4ecdffb640 1 -- 172.21.15.66:0/2711557688 >> v1:172.21.15.66:6790/0 conn(0x7f4ed01430c0 legacy=0x7f4ed014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:53.485 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.483+0000 7f4ecdffb640 1 -- 172.21.15.66:0/2711557688 >> v1:172.21.15.66:6789/0 conn(0x7f4ed013f710 legacy=0x7f4ed01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:53.485 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.483+0000 7f4ecdffb640 10 monclient: _finish_hunting 0 2024-03-25T18:14:53.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.483+0000 7f4ecdffb640 1 monclient: found mon.noname-c 2024-03-25T18:14:53.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.483+0000 7f4ecdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:53.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.483+0000 7f4ecdffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:14:53.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.483+0000 7f4ecdffb640 1 -- 172.21.15.66:0/2711557688 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed00b44c0 con 0x7f4ed00a98f0 2024-03-25T18:14:53.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.483+0000 7f4ecdffb640 10 monclient: _finish_auth 0 2024-03-25T18:14:53.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.483+0000 7f4ecdffb640 10 monclient: _check_auth_tickets 2024-03-25T18:14:53.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.483+0000 7f4ecdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:53.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.483+0000 7f4ecdffb640 1 -- 172.21.15.66:0/2711557688 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f4ec0001670 con 0x7f4ed00a98f0 2024-03-25T18:14:53.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.483+0000 7f4ecdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:53.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.483+0000 7f4ecdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:53.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.483+0000 7f4ecdffb640 10 monclient: dump: 2024-03-25T18:14:53.486 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:53.486 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:53.486 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:53.486 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:53.486 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:53.487 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:53.487 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:53.487 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:53.487 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:53.487 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:53.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.483+0000 7f4ed5940640 5 monclient: authenticate success, global_id 4717 2024-03-25T18:14:53.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.483+0000 7f4ed5940640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:53.492 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.490+0000 7f4ecdffb640 1 -- 172.21.15.66:0/2711557688 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ec00017d0 con 0x7f4ed00a98f0 2024-03-25T18:14:53.492 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.490+0000 7f4ecdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:53.492 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.490+0000 7f4ecdffb640 1 -- 172.21.15.66:0/2711557688 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f4ec0001be0 con 0x7f4ed00a98f0 2024-03-25T18:14:53.492 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.490+0000 7f4ed5940640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:53.493 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.490+0000 7f4ed5940640 4 set_mon_vals no callback set 2024-03-25T18:14:53.493 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.490+0000 7f4ecdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:53.493 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.490+0000 7f4ecdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:53.493 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.490+0000 7f4ecdffb640 10 monclient: dump: 2024-03-25T18:14:53.493 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:53.493 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:53.493 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:53.493 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:53.493 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:53.493 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:53.493 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:53.493 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:53.493 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:53.493 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:53.493 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.490+0000 7f4ed5940640 10 monclient: shutdown 2024-03-25T18:14:53.493 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.491+0000 7f4ed5940640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:53.494 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.491+0000 7f4ed5940640 1 -- 172.21.15.66:0/2711557688 >> v1:172.21.15.193:6789/0 conn(0x7f4ed00a98f0 legacy=0x7f4ed00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:53.494 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.491+0000 7f4ed5940640 1 -- 172.21.15.66:0/2711557688 shutdown_connections 2024-03-25T18:14:53.494 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.491+0000 7f4ed5940640 1 -- 172.21.15.66:0/2711557688 >> 172.21.15.66:0/2711557688 conn(0x7f4ed00b7ea0 msgr2=0x7f4ed00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:53.494 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.491+0000 7f4ed5940640 1 -- 172.21.15.66:0/2711557688 shutdown_connections 2024-03-25T18:14:53.494 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.491+0000 7f4ed5940640 1 -- 172.21.15.66:0/2711557688 wait complete. 2024-03-25T18:14:53.495 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.492+0000 7f4ed5940640 10 monclient: build_initial_monmap 2024-03-25T18:14:53.495 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.492+0000 7f4ed5940640 10 monclient: monmap: 2024-03-25T18:14:53.495 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:14:53.496 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:53.496 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:14:53.496 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:14:53.496 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:53.496 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:14:53.496 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:14:53.496 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:14:53.496 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:14:53.496 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:53.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.492+0000 7f4ed5940640 1 librados: starting msgr at 2024-03-25T18:14:53.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.492+0000 7f4ed5940640 1 librados: starting objecter 2024-03-25T18:14:53.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.492+0000 7f4ed5940640 1 Processor -- start 2024-03-25T18:14:53.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.492+0000 7f4ed5940640 1 -- start start 2024-03-25T18:14:53.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.492+0000 7f4ed5940640 1 librados: setting wanted keys 2024-03-25T18:14:53.496 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.492+0000 7f4ed5940640 1 librados: calling monclient init 2024-03-25T18:14:53.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.492+0000 7f4ed5940640 10 monclient: init 2024-03-25T18:14:53.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.493+0000 7f4ed5940640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:53.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.493+0000 7f4ed5940640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:53.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.493+0000 7f4ed5940640 10 monclient(hunting): picked mon.noname-b con 0x7f4ed00a98f0 addr v1:172.21.15.193:6789/0 2024-03-25T18:14:53.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.493+0000 7f4ed5940640 10 monclient(hunting): picked mon.noname-c con 0x7f4ed013f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:14:53.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.493+0000 7f4ed5940640 10 monclient(hunting): picked mon.noname-a con 0x7f4ed01430c0 addr v1:172.21.15.66:6789/0 2024-03-25T18:14:53.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.493+0000 7f4ed5940640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed0139330 con 0x7f4ed01430c0 2024-03-25T18:14:53.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.493+0000 7f4ed5940640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed00abe70 con 0x7f4ed00a98f0 2024-03-25T18:14:53.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.493+0000 7f4ed5940640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed00ac150 con 0x7f4ed013f710 2024-03-25T18:14:53.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.493+0000 7f4ed5940640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:53.497 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.493+0000 7f4ed5940640 10 monclient(hunting): authenticate will time out at 7642.553779s 2024-03-25T18:14:53.502 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.501+0000 7f4eceffd640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f4ed00a98f0 0x7f4ed01389d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:49634/0 (socket says 172.21.15.66:49634) 2024-03-25T18:14:53.502 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.501+0000 7f4eceffd640 1 -- 172.21.15.66:0/386548859 learned_addr learned my addr 172.21.15.66:0/386548859 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:14:53.506 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.505+0000 7f4eb7fff640 10 client.?.objecter ms_handle_connect 0x7f4ed01430c0 2024-03-25T18:14:53.506 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.505+0000 7f4eb7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:53.508 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.507+0000 7f4eb7fff640 10 client.?.objecter ms_handle_connect 0x7f4ed00a98f0 2024-03-25T18:14:53.508 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.507+0000 7f4eb7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:53.515 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.513+0000 7f4eb7fff640 1 -- 172.21.15.66:0/386548859 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291062408 0 0) 0x7f4ed00abe70 con 0x7f4ed00a98f0 2024-03-25T18:14:53.515 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.513+0000 7f4eb7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:53.515 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.513+0000 7f4eb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:53.515 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.513+0000 7f4eb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:53.515 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.513+0000 7f4eb7fff640 10 monclient(hunting): my global_id is 4720 2024-03-25T18:14:53.515 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.514+0000 7f4eb7fff640 1 -- 172.21.15.66:0/386548859 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ebc004eb0 con 0x7f4ed00a98f0 2024-03-25T18:14:53.523 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.522+0000 7f4eb7fff640 10 client.?.objecter ms_handle_connect 0x7f4ed013f710 2024-03-25T18:14:53.523 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.522+0000 7f4eb7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:53.529 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.528+0000 7f4eb7fff640 1 -- 172.21.15.66:0/386548859 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 664790655 0 0) 0x7f4ed0139330 con 0x7f4ed01430c0 2024-03-25T18:14:53.529 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.528+0000 7f4eb7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:53.529 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.528+0000 7f4eb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:53.529 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.528+0000 7f4eb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:53.529 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.528+0000 7f4eb7fff640 10 monclient(hunting): my global_id is 4725 2024-03-25T18:14:53.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.528+0000 7f4eb7fff640 1 -- 172.21.15.66:0/386548859 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ed00abe70 con 0x7f4ed01430c0 2024-03-25T18:14:53.538 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.537+0000 7f4eb7fff640 1 -- 172.21.15.66:0/386548859 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2976873486 0 0) 0x7f4ebc004eb0 con 0x7f4ed00a98f0 2024-03-25T18:14:53.538 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.537+0000 7f4eb7fff640 1 -- 172.21.15.66:0/386548859 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ed0139330 con 0x7f4ed00a98f0 2024-03-25T18:14:53.540 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.538+0000 7f4eb7fff640 1 -- 172.21.15.66:0/386548859 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f4ec00047e0 con 0x7f4ed00a98f0 2024-03-25T18:14:53.540 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.538+0000 7f4eb7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:53.540 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.538+0000 7f4eb7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:53.540 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.538+0000 7f4eb7fff640 10 monclient(hunting): dump: 2024-03-25T18:14:53.540 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:53.540 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:53.540 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:53.540 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:53.540 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:53.540 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:53.541 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:53.541 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:53.541 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:53.541 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:53.541 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.538+0000 7f4eb7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:53.567 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.565+0000 7f4eb7fff640 1 -- 172.21.15.66:0/386548859 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2486497783 0 0) 0x7f4ed00abe70 con 0x7f4ed01430c0 2024-03-25T18:14:53.567 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.565+0000 7f4eb7fff640 1 -- 172.21.15.66:0/386548859 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ebc004eb0 con 0x7f4ed01430c0 2024-03-25T18:14:53.567 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.565+0000 7f4eb7fff640 1 -- 172.21.15.66:0/386548859 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f4ec8001b50 con 0x7f4ed01430c0 2024-03-25T18:14:53.567 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.565+0000 7f4eb7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:53.567 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.565+0000 7f4eb7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:53.567 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.565+0000 7f4eb7fff640 10 monclient(hunting): dump: 2024-03-25T18:14:53.567 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:53.567 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:53.567 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:53.567 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:53.567 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:53.567 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:53.567 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:53.567 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:53.568 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:53.568 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:53.630 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.628+0000 7f4eb7fff640 1 -- 172.21.15.66:0/386548859 <== mon.0 v1:172.21.15.66:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3489995221 0 0) 0x7f4ebc004eb0 con 0x7f4ed01430c0 2024-03-25T18:14:53.630 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.628+0000 7f4eb7fff640 1 -- 172.21.15.66:0/386548859 >> v1:172.21.15.66:6790/0 conn(0x7f4ed013f710 legacy=0x7f4ed00ac570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:53.630 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.628+0000 7f4eb7fff640 1 -- 172.21.15.66:0/386548859 >> v1:172.21.15.193:6789/0 conn(0x7f4ed00a98f0 legacy=0x7f4ed01389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:53.630 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.628+0000 7f4eb7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:14:53.630 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.628+0000 7f4eb7fff640 1 monclient: found mon.a 2024-03-25T18:14:53.630 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.628+0000 7f4eb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:53.630 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.628+0000 7f4eb7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:14:53.631 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.628+0000 7f4eb7fff640 1 -- 172.21.15.66:0/386548859 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed00acc80 con 0x7f4ed01430c0 2024-03-25T18:14:53.631 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.628+0000 7f4eb7fff640 10 monclient: _finish_auth 0 2024-03-25T18:14:53.631 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.628+0000 7f4eb7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:14:53.631 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.628+0000 7f4eb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:53.631 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.628+0000 7f4ed5940640 5 monclient: authenticate success, global_id 4725 2024-03-25T18:14:53.631 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.628+0000 7f4ed5940640 10 monclient: _renew_subs 2024-03-25T18:14:53.631 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.628+0000 7f4ed5940640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:14:53.631 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.628+0000 7f4ed5940640 1 -- 172.21.15.66:0/386548859 --> v1:172.21.15.66:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed0137e30 con 0x7f4ed01430c0 2024-03-25T18:14:53.631 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.628+0000 7f4ed5940640 10 client.4725.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:53.631 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.628+0000 7f4ed5940640 10 monclient: _renew_subs 2024-03-25T18:14:53.631 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.628+0000 7f4ed5940640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:14:53.631 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.628+0000 7f4ed5940640 1 -- 172.21.15.66:0/386548859 --> v1:172.21.15.66:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed0136880 con 0x7f4ed01430c0 2024-03-25T18:14:53.631 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.628+0000 7f4ed5940640 1 librados: init done 2024-03-25T18:14:53.632 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.629+0000 7f4ed5940640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:53.632 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.629+0000 7f4ed5940640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:53.632 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.629+0000 7f4ed5940640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:14:53.632 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.629+0000 7f4ed5940640 1 -- 172.21.15.66:0/386548859 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ed0136b30 con 0x7f4ed01430c0 2024-03-25T18:14:53.947 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.945+0000 7f4eb7fff640 1 -- 172.21.15.66:0/386548859 <== mon.0 v1:172.21.15.66:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ec8001fe0 con 0x7f4ed01430c0 2024-03-25T18:14:53.947 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.945+0000 7f4eb7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:53.947 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.945+0000 7f4eb7fff640 1 -- 172.21.15.66:0/386548859 <== mon.0 v1:172.21.15.66:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f4ec8003b30 con 0x7f4ed01430c0 2024-03-25T18:14:53.947 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.945+0000 7f4eb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:53.947 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.945+0000 7f4eb7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:53.947 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.945+0000 7f4eb7fff640 10 monclient: dump: 2024-03-25T18:14:53.947 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:14:53.947 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:14:53.947 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:53.947 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:14:53.947 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:53.948 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:14:53.948 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:14:53.948 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:14:53.948 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:14:53.948 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:14:53.948 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.945+0000 7f4ecd7fa640 4 set_mon_vals no callback set 2024-03-25T18:14:53.948 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.946+0000 7f4eb7fff640 1 -- 172.21.15.66:0/386548859 <== mon.0 v1:172.21.15.66:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f4ec8003d50 con 0x7f4ed01430c0 2024-03-25T18:14:53.949 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.947+0000 7f4eceffd640 10 monclient: get_auth_request con 0x7f4ebc06de60 auth_method 0 2024-03-25T18:14:53.949 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.947+0000 7f4eb7fff640 10 client.4725.objecter ms_handle_connect 0x7f4ebc06de60 2024-03-25T18:14:53.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.953+0000 7f4eb7fff640 1 -- 172.21.15.66:0/386548859 <== mon.0 v1:172.21.15.66:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4152745027 0 0) 0x7f4ec80537e0 con 0x7f4ed01430c0 2024-03-25T18:14:53.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.953+0000 7f4eb7fff640 10 client.4725.objecter ms_dispatch 0x7f4ed0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:14:53.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.953+0000 7f4eb7fff640 3 client.4725.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:14:53.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.953+0000 7f4eb7fff640 3 client.4725.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:14:53.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.953+0000 7f4eb7fff640 20 client.4725.objecter dump_active .. 0 homeless 2024-03-25T18:14:53.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.953+0000 7f4eb7fff640 1 -- 172.21.15.66:0/386548859 <== mon.0 v1:172.21.15.66:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4ec804f2f0 con 0x7f4ed01430c0 2024-03-25T18:14:53.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.953+0000 7f4eb7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:53.955 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:53.953+0000 7f4eb7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:54.223 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:54.221+0000 7f4ed5940640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:54.223 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:54.221+0000 7f4ed5940640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:54.223 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:54.221+0000 7f4ed5940640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:14:54.223 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:54.221+0000 7f4ed5940640 1 -- 172.21.15.66:0/386548859 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ed00627b0 con 0x7f4ed01430c0 2024-03-25T18:14:54.261 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:54.260+0000 7f4eb7fff640 1 -- 172.21.15.66:0/386548859 <== mon.0 v1:172.21.15.66:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4258808387) 0x7f4ec8051220 con 0x7f4ed01430c0 2024-03-25T18:14:54.261 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:54.260+0000 7f4eb7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:54.261 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:54.260+0000 7f4eb7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:54.262 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:14:54.262 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":11,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:13:50.353085+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:50.353085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:54.264 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:54.262+0000 7f4ed5940640 10 librados: watch_flush enter 2024-03-25T18:14:54.264 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:54.262+0000 7f4ed5940640 10 librados: watch_flush exit 2024-03-25T18:14:54.264 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:54.262+0000 7f4ed5940640 20 client.4725.objecter shutdown clearing up homeless session... 2024-03-25T18:14:54.264 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:54.262+0000 7f4ed5940640 10 client.4725.objecter successfully canceled tick 2024-03-25T18:14:54.265 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:54.262+0000 7f4ed5940640 1 -- 172.21.15.66:0/386548859 >> v1:172.21.15.193:6800/2275676803 conn(0x7f4ebc06de60 legacy=0x7f4ebc070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:54.265 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:54.262+0000 7f4ed5940640 10 monclient: shutdown 2024-03-25T18:14:54.265 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:54.262+0000 7f4ed5940640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:54.265 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:54.262+0000 7f4ed5940640 1 -- 172.21.15.66:0/386548859 >> v1:172.21.15.66:6789/0 conn(0x7f4ed01430c0 legacy=0x7f4ed01e2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:54.265 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:54.262+0000 7f4ed5940640 1 -- 172.21.15.66:0/386548859 shutdown_connections 2024-03-25T18:14:54.265 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:54.262+0000 7f4ed5940640 1 -- 172.21.15.66:0/386548859 >> 172.21.15.66:0/386548859 conn(0x7f4ed00b7ea0 msgr2=0x7f4ed0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:54.265 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:54.263+0000 7f4ed5940640 1 -- 172.21.15.66:0/386548859 shutdown_connections 2024-03-25T18:14:54.265 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:54.263+0000 7f4ed5940640 1 -- 172.21.15.66:0/386548859 wait complete. 2024-03-25T18:14:54.265 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:14:54.263+0000 7f4ed5940640 1 librados: shutdown 2024-03-25T18:14:54.274 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:55.763 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:11:26.743 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:55.771 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:11:26.752 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:55.806 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:11:26.787 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:55.814 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:11:26.772 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:55.817 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:11:26.797 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:55.830 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:11:26.788 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:55.831 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:11:26.811 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:55.848 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:11:26.807 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:55.857 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:11:26.837 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:01.081 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:01.198 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.197+0000 7f3947c50640 10 monclient: get_monmap_and_config 2024-03-25T18:15:01.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.197+0000 7f3947c50640 10 monclient: build_initial_monmap 2024-03-25T18:15:01.199 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.197+0000 7f3947c50640 10 monclient: monmap: 2024-03-25T18:15:01.199 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:15:01.199 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:15:01.199 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:15:01.198703+0000 2024-03-25T18:15:01.199 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:15:01.198703+0000 2024-03-25T18:15:01.199 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:01.199 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:15:01.199 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:15:01.199 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:15:01.199 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:15:01.199 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:15:01.200 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.197+0000 7f3947c50640 1 Processor -- start 2024-03-25T18:15:01.200 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.198+0000 7f3947c50640 1 -- start start 2024-03-25T18:15:01.200 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.198+0000 7f3947c50640 10 monclient: init 2024-03-25T18:15:01.200 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.198+0000 7f3947c50640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:01.200 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.198+0000 7f3947c50640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:01.200 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.198+0000 7f3947c50640 10 monclient(hunting): picked mon.noname-b con 0x7f39400a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:15:01.200 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.198+0000 7f3947c50640 10 monclient(hunting): picked mon.noname-a con 0x7f394013f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:15:01.201 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.198+0000 7f3947c50640 10 monclient(hunting): picked mon.noname-c con 0x7f39401430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:15:01.201 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.199+0000 7f3947c50640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3940142520 con 0x7f394013f710 2024-03-25T18:15:01.201 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.199+0000 7f3947c50640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39400b28a0 con 0x7f39401430c0 2024-03-25T18:15:01.201 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.199+0000 7f3947c50640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39400b2a30 con 0x7f39400a98f0 2024-03-25T18:15:01.201 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.199+0000 7f3947c50640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:01.201 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.199+0000 7f3947c50640 10 monclient(hunting): authenticate will time out at 7650.259123s 2024-03-25T18:15:01.208 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.207+0000 7f39459ec640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f39400a98f0 0x7f39400a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:37602/0 (socket says 172.21.15.66:37602) 2024-03-25T18:15:01.208 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.207+0000 7f39459ec640 1 -- 172.21.15.66:0/3730723436 learned_addr learned my addr 172.21.15.66:0/3730723436 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:15:01.219 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.216+0000 7f39449ea640 1 -- 172.21.15.66:0/3730723436 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2602687512 0 0) 0x7f39400b2a30 con 0x7f39400a98f0 2024-03-25T18:15:01.219 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.216+0000 7f39449ea640 10 monclient(hunting): _negotiate 2024-03-25T18:15:01.219 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.216+0000 7f39449ea640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:01.219 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.216+0000 7f39449ea640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:01.219 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.216+0000 7f39449ea640 10 monclient(hunting): my global_id is 4721 2024-03-25T18:15:01.219 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.216+0000 7f39449ea640 1 -- 172.21.15.66:0/3730723436 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f392c003520 con 0x7f39400a98f0 2024-03-25T18:15:01.219 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.217+0000 7f39449ea640 1 -- 172.21.15.66:0/3730723436 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3085979748 0 0) 0x7f39400b28a0 con 0x7f39401430c0 2024-03-25T18:15:01.219 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.217+0000 7f39449ea640 10 monclient(hunting): _negotiate 2024-03-25T18:15:01.219 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.217+0000 7f39449ea640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:01.219 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.217+0000 7f39449ea640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:01.219 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.217+0000 7f39449ea640 10 monclient(hunting): my global_id is 4723 2024-03-25T18:15:01.219 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.217+0000 7f39449ea640 1 -- 172.21.15.66:0/3730723436 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39400b2a30 con 0x7f39401430c0 2024-03-25T18:15:01.222 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.220+0000 7f39449ea640 1 -- 172.21.15.66:0/3730723436 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1347429252 0 0) 0x7f3940142520 con 0x7f394013f710 2024-03-25T18:15:01.222 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.220+0000 7f39449ea640 10 monclient(hunting): _negotiate 2024-03-25T18:15:01.222 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.220+0000 7f39449ea640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:01.222 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.220+0000 7f39449ea640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:01.222 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.220+0000 7f39449ea640 10 monclient(hunting): my global_id is 4728 2024-03-25T18:15:01.222 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.220+0000 7f39449ea640 1 -- 172.21.15.66:0/3730723436 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39400b28a0 con 0x7f394013f710 2024-03-25T18:15:01.225 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.223+0000 7f39449ea640 1 -- 172.21.15.66:0/3730723436 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3243311029 0 0) 0x7f39400b2a30 con 0x7f39401430c0 2024-03-25T18:15:01.225 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.223+0000 7f39449ea640 1 -- 172.21.15.66:0/3730723436 >> v1:172.21.15.66:6790/0 conn(0x7f39400a98f0 legacy=0x7f39400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:01.225 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.223+0000 7f39449ea640 1 -- 172.21.15.66:0/3730723436 >> v1:172.21.15.66:6789/0 conn(0x7f394013f710 legacy=0x7f39401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:01.225 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.223+0000 7f39449ea640 10 monclient: _finish_hunting 0 2024-03-25T18:15:01.225 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.223+0000 7f39449ea640 1 monclient: found mon.noname-c 2024-03-25T18:15:01.225 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.223+0000 7f39449ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:01.225 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.223+0000 7f39449ea640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:15:01.225 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.223+0000 7f39449ea640 1 -- 172.21.15.66:0/3730723436 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39400b44c0 con 0x7f39401430c0 2024-03-25T18:15:01.226 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.223+0000 7f39449ea640 10 monclient: _finish_auth 0 2024-03-25T18:15:01.226 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.223+0000 7f39449ea640 10 monclient: _check_auth_tickets 2024-03-25T18:15:01.226 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.223+0000 7f39449ea640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:01.226 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.223+0000 7f39449ea640 1 -- 172.21.15.66:0/3730723436 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f39380019f0 con 0x7f39401430c0 2024-03-25T18:15:01.226 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.223+0000 7f3947c50640 5 monclient: authenticate success, global_id 4723 2024-03-25T18:15:01.226 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.223+0000 7f3947c50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:01.226 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.223+0000 7f39449ea640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:01.226 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.223+0000 7f39449ea640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:01.226 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.223+0000 7f39449ea640 10 monclient: dump: 2024-03-25T18:15:01.226 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:15:01.226 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:15:01.226 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:15:01.226 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:15:01.227 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:01.227 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:15:01.227 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:15:01.227 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:15:01.227 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:15:01.227 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:15:01.227 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.223+0000 7f3947c50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:01.228 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.227+0000 7f39449ea640 1 -- 172.21.15.66:0/3730723436 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3938001b50 con 0x7f39401430c0 2024-03-25T18:15:01.229 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.227+0000 7f39449ea640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:01.229 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.227+0000 7f39449ea640 1 -- 172.21.15.66:0/3730723436 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f3938001f60 con 0x7f39401430c0 2024-03-25T18:15:01.229 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.227+0000 7f3947c50640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:01.229 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.227+0000 7f3947c50640 4 set_mon_vals no callback set 2024-03-25T18:15:01.229 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.227+0000 7f39449ea640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:01.229 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.227+0000 7f39449ea640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:01.229 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.227+0000 7f39449ea640 10 monclient: dump: 2024-03-25T18:15:01.229 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:15:01.229 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:15:01.229 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:15:01.229 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:15:01.229 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:01.229 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:15:01.229 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:15:01.230 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:15:01.230 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:15:01.230 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:15:01.231 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.228+0000 7f3947c50640 10 monclient: shutdown 2024-03-25T18:15:01.231 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.228+0000 7f3947c50640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:01.231 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.228+0000 7f3947c50640 1 -- 172.21.15.66:0/3730723436 >> v1:172.21.15.193:6789/0 conn(0x7f39401430c0 legacy=0x7f394014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:01.231 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.228+0000 7f3947c50640 1 -- 172.21.15.66:0/3730723436 shutdown_connections 2024-03-25T18:15:01.231 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.228+0000 7f3947c50640 1 -- 172.21.15.66:0/3730723436 >> 172.21.15.66:0/3730723436 conn(0x7f39400b7ea0 msgr2=0x7f39400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:01.232 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.228+0000 7f3947c50640 1 -- 172.21.15.66:0/3730723436 shutdown_connections 2024-03-25T18:15:01.232 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.228+0000 7f3947c50640 1 -- 172.21.15.66:0/3730723436 wait complete. 2024-03-25T18:15:01.232 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.229+0000 7f3947c50640 10 monclient: build_initial_monmap 2024-03-25T18:15:01.232 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.229+0000 7f3947c50640 10 monclient: monmap: 2024-03-25T18:15:01.232 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:15:01.232 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:01.232 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:15:01.232 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:15:01.232 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:01.232 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:15:01.232 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:15:01.232 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:15:01.232 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:15:01.232 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:15:01.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.229+0000 7f3947c50640 1 librados: starting msgr at 2024-03-25T18:15:01.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.229+0000 7f3947c50640 1 librados: starting objecter 2024-03-25T18:15:01.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.229+0000 7f3947c50640 1 Processor -- start 2024-03-25T18:15:01.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.229+0000 7f3947c50640 1 -- start start 2024-03-25T18:15:01.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.229+0000 7f3947c50640 1 librados: setting wanted keys 2024-03-25T18:15:01.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.229+0000 7f3947c50640 1 librados: calling monclient init 2024-03-25T18:15:01.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.229+0000 7f3947c50640 10 monclient: init 2024-03-25T18:15:01.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.230+0000 7f3947c50640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:01.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.230+0000 7f3947c50640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:01.233 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.230+0000 7f3947c50640 10 monclient(hunting): picked mon.noname-c con 0x7f39400a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:15:01.234 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.230+0000 7f3947c50640 10 monclient(hunting): picked mon.noname-a con 0x7f394013f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:15:01.234 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.230+0000 7f3947c50640 10 monclient(hunting): picked mon.noname-b con 0x7f39401430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:15:01.234 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.230+0000 7f3947c50640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f394013c190 con 0x7f394013f710 2024-03-25T18:15:01.234 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.230+0000 7f3947c50640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39400b7cf0 con 0x7f39401430c0 2024-03-25T18:15:01.234 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.230+0000 7f3947c50640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f394013ad50 con 0x7f39400a98f0 2024-03-25T18:15:01.234 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.230+0000 7f3947c50640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:01.234 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.230+0000 7f3947c50640 10 monclient(hunting): authenticate will time out at 7650.290681s 2024-03-25T18:15:01.238 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.236+0000 7f39461ed640 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f39401430c0 0x7f39401e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.66:35142/0 (socket says 172.21.15.66:35142) 2024-03-25T18:15:01.238 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.236+0000 7f39461ed640 1 -- 172.21.15.66:0/3414722705 learned_addr learned my addr 172.21.15.66:0/3414722705 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:15:01.245 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.243+0000 7f39367fc640 10 client.?.objecter ms_handle_connect 0x7f39401430c0 2024-03-25T18:15:01.245 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.243+0000 7f39367fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:01.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.252+0000 7f39367fc640 1 -- 172.21.15.66:0/3414722705 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 647376479 0 0) 0x7f39400b7cf0 con 0x7f39401430c0 2024-03-25T18:15:01.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.252+0000 7f39367fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:01.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.252+0000 7f39367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:01.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.252+0000 7f39367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:01.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.252+0000 7f39367fc640 10 monclient(hunting): my global_id is 4726 2024-03-25T18:15:01.253 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.252+0000 7f39367fc640 1 -- 172.21.15.66:0/3414722705 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3924004eb0 con 0x7f39401430c0 2024-03-25T18:15:01.257 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.256+0000 7f39367fc640 10 client.?.objecter ms_handle_connect 0x7f39400a98f0 2024-03-25T18:15:01.258 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.256+0000 7f39367fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:01.260 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.259+0000 7f39367fc640 10 client.?.objecter ms_handle_connect 0x7f394013f710 2024-03-25T18:15:01.261 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.259+0000 7f39367fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:01.267 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.265+0000 7f39367fc640 1 -- 172.21.15.66:0/3414722705 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 78515573 0 0) 0x7f394013ad50 con 0x7f39400a98f0 2024-03-25T18:15:01.267 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.265+0000 7f39367fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:01.267 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.265+0000 7f39367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:01.267 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.265+0000 7f39367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:01.267 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.265+0000 7f39367fc640 10 monclient(hunting): my global_id is 4724 2024-03-25T18:15:01.267 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.265+0000 7f39367fc640 1 -- 172.21.15.66:0/3414722705 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39400b7cf0 con 0x7f39400a98f0 2024-03-25T18:15:01.273 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.272+0000 7f39367fc640 1 -- 172.21.15.66:0/3414722705 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3981858102 0 0) 0x7f39400b28a0 con 0x7f394013f710 2024-03-25T18:15:01.273 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.272+0000 7f39367fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:01.273 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.272+0000 7f39367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:01.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.272+0000 7f39367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:01.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.272+0000 7f39367fc640 10 monclient(hunting): my global_id is 4731 2024-03-25T18:15:01.274 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.272+0000 7f39367fc640 1 -- 172.21.15.66:0/3414722705 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f394013ad50 con 0x7f394013f710 2024-03-25T18:15:01.277 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.275+0000 7f39367fc640 1 -- 172.21.15.66:0/3414722705 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 282398605 0 0) 0x7f39400b7cf0 con 0x7f39400a98f0 2024-03-25T18:15:01.277 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.275+0000 7f39367fc640 1 -- 172.21.15.66:0/3414722705 --> v1:172.21.15.66:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39400b28a0 con 0x7f39400a98f0 2024-03-25T18:15:01.277 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.275+0000 7f39367fc640 1 -- 172.21.15.66:0/3414722705 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f393c001950 con 0x7f39400a98f0 2024-03-25T18:15:01.277 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.275+0000 7f39367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:01.277 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.275+0000 7f39367fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:01.277 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.275+0000 7f39367fc640 10 monclient(hunting): dump: 2024-03-25T18:15:01.277 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:15:01.277 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:15:01.277 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:15:01.277 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:15:01.277 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:01.278 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:15:01.278 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:15:01.278 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:15:01.278 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:15:01.278 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:15:01.278 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.275+0000 7f39367fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:01.279 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.277+0000 7f39367fc640 1 -- 172.21.15.66:0/3414722705 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1398249218 0 0) 0x7f3924004eb0 con 0x7f39401430c0 2024-03-25T18:15:01.279 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.278+0000 7f39367fc640 1 -- 172.21.15.66:0/3414722705 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39400b7cf0 con 0x7f39401430c0 2024-03-25T18:15:01.279 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.278+0000 7f39367fc640 1 -- 172.21.15.66:0/3414722705 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f3938001fa0 con 0x7f39401430c0 2024-03-25T18:15:01.279 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.278+0000 7f39367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:01.279 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.278+0000 7f39367fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:01.279 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.278+0000 7f39367fc640 10 monclient(hunting): dump: 2024-03-25T18:15:01.280 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:15:01.280 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:15:01.280 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:15:01.280 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:15:01.280 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:01.280 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:15:01.280 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:15:01.280 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:15:01.280 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:15:01.280 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:15:01.285 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.283+0000 7f39367fc640 1 -- 172.21.15.66:0/3414722705 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4291486822 0 0) 0x7f39400b7cf0 con 0x7f39401430c0 2024-03-25T18:15:01.285 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.283+0000 7f39367fc640 1 -- 172.21.15.66:0/3414722705 >> v1:172.21.15.66:6790/0 conn(0x7f39400a98f0 legacy=0x7f394013ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:01.285 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.283+0000 7f39367fc640 1 -- 172.21.15.66:0/3414722705 >> v1:172.21.15.66:6789/0 conn(0x7f394013f710 legacy=0x7f39401df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:01.286 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.283+0000 7f39367fc640 10 monclient: _finish_hunting 0 2024-03-25T18:15:01.286 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.283+0000 7f39367fc640 1 monclient: found mon.b 2024-03-25T18:15:01.286 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.283+0000 7f39367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:01.286 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.283+0000 7f39367fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:15:01.286 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.283+0000 7f39367fc640 1 -- 172.21.15.66:0/3414722705 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f394013afc0 con 0x7f39401430c0 2024-03-25T18:15:01.286 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.283+0000 7f39367fc640 10 monclient: _finish_auth 0 2024-03-25T18:15:01.286 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.283+0000 7f39367fc640 10 monclient: _check_auth_tickets 2024-03-25T18:15:01.286 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.283+0000 7f39367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:01.286 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.283+0000 7f3947c50640 5 monclient: authenticate success, global_id 4726 2024-03-25T18:15:01.286 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.283+0000 7f3947c50640 10 monclient: _renew_subs 2024-03-25T18:15:01.286 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.283+0000 7f3947c50640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:15:01.286 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.283+0000 7f3947c50640 1 -- 172.21.15.66:0/3414722705 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f394013b2a0 con 0x7f39401430c0 2024-03-25T18:15:01.286 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.284+0000 7f3947c50640 10 client.4726.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:01.287 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.284+0000 7f3947c50640 10 monclient: _renew_subs 2024-03-25T18:15:01.287 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.284+0000 7f3947c50640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:15:01.287 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.284+0000 7f3947c50640 1 -- 172.21.15.66:0/3414722705 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f394012fa90 con 0x7f39401430c0 2024-03-25T18:15:01.287 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.284+0000 7f3947c50640 1 librados: init done 2024-03-25T18:15:01.287 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.285+0000 7f3947c50640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:01.287 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.285+0000 7f3947c50640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:01.287 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.285+0000 7f3947c50640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:15:01.288 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.285+0000 7f3947c50640 1 -- 172.21.15.66:0/3414722705 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39400b3ef0 con 0x7f39401430c0 2024-03-25T18:15:01.584 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.582+0000 7f39367fc640 1 -- 172.21.15.66:0/3414722705 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3938003c20 con 0x7f39401430c0 2024-03-25T18:15:01.584 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.582+0000 7f39367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:01.584 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.582+0000 7f39367fc640 1 -- 172.21.15.66:0/3414722705 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f3938003f70 con 0x7f39401430c0 2024-03-25T18:15:01.584 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.582+0000 7f39367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:01.584 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.582+0000 7f39367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:01.584 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.582+0000 7f39367fc640 10 monclient: dump: 2024-03-25T18:15:01.584 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:15:01.584 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:15:01.584 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:15:01.584 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:15:01.584 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:01.584 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:15:01.584 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:15:01.585 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:15:01.585 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:15:01.585 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:15:01.585 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.582+0000 7f3937fff640 4 set_mon_vals no callback set 2024-03-25T18:15:01.585 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.583+0000 7f39367fc640 1 -- 172.21.15.66:0/3414722705 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f3938004190 con 0x7f39401430c0 2024-03-25T18:15:01.585 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.583+0000 7f39459ec640 10 monclient: get_auth_request con 0x7f392406dfb0 auth_method 0 2024-03-25T18:15:01.586 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.584+0000 7f39367fc640 10 client.4726.objecter ms_handle_connect 0x7f392406dfb0 2024-03-25T18:15:01.590 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.588+0000 7f39367fc640 1 -- 172.21.15.66:0/3414722705 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4152745027 0 0) 0x7f3938054b60 con 0x7f39401430c0 2024-03-25T18:15:01.590 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.588+0000 7f39367fc640 10 client.4726.objecter ms_dispatch 0x7f3940000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:15:01.590 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.588+0000 7f39367fc640 3 client.4726.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:15:01.590 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.588+0000 7f39367fc640 3 client.4726.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:15:01.590 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.589+0000 7f39367fc640 20 client.4726.objecter dump_active .. 0 homeless 2024-03-25T18:15:01.591 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.589+0000 7f39367fc640 1 -- 172.21.15.66:0/3414722705 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3938050670 con 0x7f39401430c0 2024-03-25T18:15:01.591 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.589+0000 7f39367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:01.591 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.589+0000 7f39367fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:01.809 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.807+0000 7f3947c50640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:01.809 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.807+0000 7f3947c50640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:01.809 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.807+0000 7f3947c50640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:15:01.809 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.807+0000 7f3947c50640 1 -- 172.21.15.66:0/3414722705 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39400627b0 con 0x7f39401430c0 2024-03-25T18:15:01.835 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.833+0000 7f39367fc640 1 -- 172.21.15.66:0/3414722705 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4258808387) 0x7f39380525a0 con 0x7f39401430c0 2024-03-25T18:15:01.835 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.833+0000 7f39367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:01.835 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.833+0000 7f39367fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:01.835 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:15:01.835 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":11,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:13:50.353085+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:50.353085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:01.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.835+0000 7f3947c50640 10 librados: watch_flush enter 2024-03-25T18:15:01.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.835+0000 7f3947c50640 10 librados: watch_flush exit 2024-03-25T18:15:01.837 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.835+0000 7f3947c50640 20 client.4726.objecter shutdown clearing up homeless session... 2024-03-25T18:15:01.838 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.835+0000 7f3947c50640 10 client.4726.objecter successfully canceled tick 2024-03-25T18:15:01.838 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.835+0000 7f3947c50640 1 -- 172.21.15.66:0/3414722705 >> v1:172.21.15.193:6800/2275676803 conn(0x7f392406dfb0 legacy=0x7f3924070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:01.838 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.835+0000 7f3947c50640 10 monclient: shutdown 2024-03-25T18:15:01.838 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.835+0000 7f3947c50640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:01.838 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.835+0000 7f3947c50640 1 -- 172.21.15.66:0/3414722705 >> v1:172.21.15.193:6789/0 conn(0x7f39401430c0 legacy=0x7f39401e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:01.838 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.835+0000 7f3947c50640 1 -- 172.21.15.66:0/3414722705 shutdown_connections 2024-03-25T18:15:01.838 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.835+0000 7f3947c50640 1 -- 172.21.15.66:0/3414722705 >> 172.21.15.66:0/3414722705 conn(0x7f39400b7ea0 msgr2=0x7f3940074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:01.838 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.835+0000 7f3947c50640 1 -- 172.21.15.66:0/3414722705 shutdown_connections 2024-03-25T18:15:01.838 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.835+0000 7f3947c50640 1 -- 172.21.15.66:0/3414722705 wait complete. 2024-03-25T18:15:01.838 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:01.835+0000 7f3947c50640 1 librados: shutdown 2024-03-25T18:15:01.849 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:08.656 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:08.768 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.766+0000 7f8fbec51640 10 monclient: get_monmap_and_config 2024-03-25T18:15:08.768 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.766+0000 7f8fbec51640 10 monclient: build_initial_monmap 2024-03-25T18:15:08.768 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.767+0000 7f8fbec51640 10 monclient: monmap: 2024-03-25T18:15:08.768 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:15:08.768 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:15:08.768 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:15:08.768505+0000 2024-03-25T18:15:08.769 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:15:08.768505+0000 2024-03-25T18:15:08.769 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:08.769 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:15:08.769 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:15:08.769 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:15:08.769 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:15:08.769 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:15:08.769 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.768+0000 7f8fbec51640 1 Processor -- start 2024-03-25T18:15:08.770 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.768+0000 7f8fbec51640 1 -- start start 2024-03-25T18:15:08.770 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.768+0000 7f8fbec51640 10 monclient: init 2024-03-25T18:15:08.770 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.769+0000 7f8fbec51640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:08.770 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.769+0000 7f8fbec51640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:15:08.770 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.769+0000 7f8fbec51640 10 monclient(hunting): picked mon.noname-a con 0x7f8fb80a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:15:08.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.769+0000 7f8fbec51640 10 monclient(hunting): picked mon.noname-b con 0x7f8fb813f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:15:08.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.769+0000 7f8fbec51640 10 monclient(hunting): picked mon.noname-c con 0x7f8fb81430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:15:08.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.769+0000 7f8fbec51640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fb8142520 con 0x7f8fb80a98f0 2024-03-25T18:15:08.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.769+0000 7f8fbec51640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fb80b28a0 con 0x7f8fb81430c0 2024-03-25T18:15:08.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.769+0000 7f8fbec51640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fb80b2a30 con 0x7f8fb813f710 2024-03-25T18:15:08.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.769+0000 7f8fbec51640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:08.771 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.769+0000 7f8fbec51640 10 monclient(hunting): authenticate will time out at 7657.829434s 2024-03-25T18:15:08.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.783+0000 7f8fbc9ed640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f8fb80a98f0 0x7f8fb80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:54432/0 (socket says 172.21.15.66:54432) 2024-03-25T18:15:08.784 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.783+0000 7f8fbc9ed640 1 -- 172.21.15.66:0/3641986563 learned_addr learned my addr 172.21.15.66:0/3641986563 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:15:08.797 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.795+0000 7f8fb77fe640 1 -- 172.21.15.66:0/3641986563 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177205217 0 0) 0x7f8fb80b2a30 con 0x7f8fb813f710 2024-03-25T18:15:08.797 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.795+0000 7f8fb77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:08.797 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.795+0000 7f8fb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:08.797 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.795+0000 7f8fb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:08.797 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.795+0000 7f8fb77fe640 10 monclient(hunting): my global_id is 4727 2024-03-25T18:15:08.797 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.795+0000 7f8fb77fe640 1 -- 172.21.15.66:0/3641986563 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fa8003520 con 0x7f8fb813f710 2024-03-25T18:15:08.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.798+0000 7f8fb77fe640 1 -- 172.21.15.66:0/3641986563 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2324747051 0 0) 0x7f8fb8142520 con 0x7f8fb80a98f0 2024-03-25T18:15:08.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.798+0000 7f8fb77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:08.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.798+0000 7f8fb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:08.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.798+0000 7f8fb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:08.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.798+0000 7f8fb77fe640 10 monclient(hunting): my global_id is 4734 2024-03-25T18:15:08.799 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.798+0000 7f8fb77fe640 1 -- 172.21.15.66:0/3641986563 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fb80b2a30 con 0x7f8fb80a98f0 2024-03-25T18:15:08.806 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.804+0000 7f8fb77fe640 1 -- 172.21.15.66:0/3641986563 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3121034937 0 0) 0x7f8fb80b28a0 con 0x7f8fb81430c0 2024-03-25T18:15:08.806 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.804+0000 7f8fb77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:08.806 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.804+0000 7f8fb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:08.806 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.804+0000 7f8fb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:08.806 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.804+0000 7f8fb77fe640 10 monclient(hunting): my global_id is 4729 2024-03-25T18:15:08.806 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.804+0000 7f8fb77fe640 1 -- 172.21.15.66:0/3641986563 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fb8142520 con 0x7f8fb81430c0 2024-03-25T18:15:08.812 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.811+0000 7f8fb77fe640 1 -- 172.21.15.66:0/3641986563 <== mon.2 v1:172.21.15.66:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2722170396 0 0) 0x7f8fa8003520 con 0x7f8fb813f710 2024-03-25T18:15:08.813 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.811+0000 7f8fb77fe640 1 -- 172.21.15.66:0/3641986563 >> v1:172.21.15.193:6789/0 conn(0x7f8fb81430c0 legacy=0x7f8fb814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:08.813 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.811+0000 7f8fb77fe640 1 -- 172.21.15.66:0/3641986563 >> v1:172.21.15.66:6789/0 conn(0x7f8fb80a98f0 legacy=0x7f8fb80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:08.813 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.811+0000 7f8fb77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:15:08.813 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.811+0000 7f8fb77fe640 1 monclient: found mon.noname-b 2024-03-25T18:15:08.813 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.811+0000 7f8fb77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:08.813 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.811+0000 7f8fb77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.66:6790/0 2024-03-25T18:15:08.813 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.811+0000 7f8fb77fe640 1 -- 172.21.15.66:0/3641986563 --> v1:172.21.15.66:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb80b44c0 con 0x7f8fb813f710 2024-03-25T18:15:08.813 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.811+0000 7f8fb77fe640 10 monclient: _finish_auth 0 2024-03-25T18:15:08.813 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.811+0000 7f8fb77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:15:08.813 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.811+0000 7f8fb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:08.813 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.811+0000 7f8fb77fe640 1 -- 172.21.15.66:0/3641986563 <== mon.2 v1:172.21.15.66:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8fa4001690 con 0x7f8fb813f710 2024-03-25T18:15:08.813 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.811+0000 7f8fbec51640 5 monclient: authenticate success, global_id 4727 2024-03-25T18:15:08.813 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.811+0000 7f8fbec51640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:08.813 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.811+0000 7f8fb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:08.814 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.811+0000 7f8fb77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:08.814 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.811+0000 7f8fb77fe640 10 monclient: dump: 2024-03-25T18:15:08.814 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:15:08.814 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:15:08.814 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:15:08.814 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:15:08.814 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:08.814 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:15:08.814 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:15:08.814 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:15:08.814 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:15:08.814 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:15:08.814 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.811+0000 7f8fbec51640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:08.824 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.823+0000 7f8fb77fe640 1 -- 172.21.15.66:0/3641986563 <== mon.2 v1:172.21.15.66:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8fa40017f0 con 0x7f8fb813f710 2024-03-25T18:15:08.824 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.823+0000 7f8fb77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:08.824 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.823+0000 7f8fb77fe640 1 -- 172.21.15.66:0/3641986563 <== mon.2 v1:172.21.15.66:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8fa4001c00 con 0x7f8fb813f710 2024-03-25T18:15:08.824 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.823+0000 7f8fbec51640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:08.824 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.823+0000 7f8fbec51640 4 set_mon_vals no callback set 2024-03-25T18:15:08.824 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.823+0000 7f8fb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:08.824 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.823+0000 7f8fb77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:08.825 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.823+0000 7f8fb77fe640 10 monclient: dump: 2024-03-25T18:15:08.825 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:15:08.825 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:15:08.825 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:15:08.825 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:15:08.825 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:08.825 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:15:08.825 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:15:08.825 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:15:08.825 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:15:08.825 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:15:08.826 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.823+0000 7f8fbec51640 10 monclient: shutdown 2024-03-25T18:15:08.826 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.823+0000 7f8fbec51640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:08.826 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.823+0000 7f8fbec51640 1 -- 172.21.15.66:0/3641986563 >> v1:172.21.15.66:6790/0 conn(0x7f8fb813f710 legacy=0x7f8fb81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:08.827 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.823+0000 7f8fbec51640 1 -- 172.21.15.66:0/3641986563 shutdown_connections 2024-03-25T18:15:08.827 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.823+0000 7f8fbec51640 1 -- 172.21.15.66:0/3641986563 >> 172.21.15.66:0/3641986563 conn(0x7f8fb80b7ea0 msgr2=0x7f8fb80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:08.827 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.824+0000 7f8fbec51640 1 -- 172.21.15.66:0/3641986563 shutdown_connections 2024-03-25T18:15:08.827 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.824+0000 7f8fbec51640 1 -- 172.21.15.66:0/3641986563 wait complete. 2024-03-25T18:15:08.827 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.824+0000 7f8fbec51640 10 monclient: build_initial_monmap 2024-03-25T18:15:08.827 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.824+0000 7f8fbec51640 10 monclient: monmap: 2024-03-25T18:15:08.827 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:15:08.827 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:08.827 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:15:08.827 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:15:08.827 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:08.827 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:15:08.827 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:15:08.827 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:15:08.828 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:15:08.828 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:15:08.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.825+0000 7f8fbec51640 1 librados: starting msgr at 2024-03-25T18:15:08.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.825+0000 7f8fbec51640 1 librados: starting objecter 2024-03-25T18:15:08.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.825+0000 7f8fbec51640 1 Processor -- start 2024-03-25T18:15:08.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.825+0000 7f8fbec51640 1 -- start start 2024-03-25T18:15:08.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.825+0000 7f8fbec51640 1 librados: setting wanted keys 2024-03-25T18:15:08.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.825+0000 7f8fbec51640 1 librados: calling monclient init 2024-03-25T18:15:08.828 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.825+0000 7f8fbec51640 10 monclient: init 2024-03-25T18:15:08.829 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.825+0000 7f8fbec51640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:08.829 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.825+0000 7f8fbec51640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:15:08.829 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.826+0000 7f8fbec51640 10 monclient(hunting): picked mon.noname-a con 0x7f8fb80a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:15:08.829 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.826+0000 7f8fbec51640 10 monclient(hunting): picked mon.noname-c con 0x7f8fb813f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:15:08.829 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.826+0000 7f8fbec51640 10 monclient(hunting): picked mon.noname-b con 0x7f8fb81430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:15:08.829 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.826+0000 7f8fbec51640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fb80b1870 con 0x7f8fb80a98f0 2024-03-25T18:15:08.829 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.826+0000 7f8fbec51640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fb80ac910 con 0x7f8fb81430c0 2024-03-25T18:15:08.829 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.826+0000 7f8fbec51640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fb80abb00 con 0x7f8fb813f710 2024-03-25T18:15:08.829 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.826+0000 7f8fbec51640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:08.829 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.826+0000 7f8fbec51640 10 monclient(hunting): authenticate will time out at 7657.886252s 2024-03-25T18:15:08.832 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.831+0000 7f8fbc9ed640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f8fb80a98f0 0x7f8fb8138670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:54464/0 (socket says 172.21.15.66:54464) 2024-03-25T18:15:08.832 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.831+0000 7f8fbc9ed640 1 -- 172.21.15.66:0/67681916 learned_addr learned my addr 172.21.15.66:0/67681916 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:15:08.838 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.836+0000 7f8fb57fa640 10 client.?.objecter ms_handle_connect 0x7f8fb80a98f0 2024-03-25T18:15:08.838 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.836+0000 7f8fb57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:08.844 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.842+0000 7f8fb57fa640 1 -- 172.21.15.66:0/67681916 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4279065361 0 0) 0x7f8fb80b2a30 con 0x7f8fb80a98f0 2024-03-25T18:15:08.844 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.842+0000 7f8fb57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:08.844 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.842+0000 7f8fb57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:08.844 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.842+0000 7f8fb57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:08.844 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.842+0000 7f8fb57fa640 10 monclient(hunting): my global_id is 4737 2024-03-25T18:15:08.844 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.842+0000 7f8fb57fa640 1 -- 172.21.15.66:0/67681916 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f9c004eb0 con 0x7f8fb80a98f0 2024-03-25T18:15:08.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.843+0000 7f8fb57fa640 10 client.?.objecter ms_handle_connect 0x7f8fb81430c0 2024-03-25T18:15:08.845 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.843+0000 7f8fb57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:08.848 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.847+0000 7f8fb57fa640 10 client.?.objecter ms_handle_connect 0x7f8fb813f710 2024-03-25T18:15:08.848 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.847+0000 7f8fb57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:08.858 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.857+0000 7f8fb57fa640 1 -- 172.21.15.66:0/67681916 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 683672666 0 0) 0x7f8f9c004eb0 con 0x7f8fb80a98f0 2024-03-25T18:15:08.858 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.857+0000 7f8fb57fa640 1 -- 172.21.15.66:0/67681916 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fb80b2a30 con 0x7f8fb80a98f0 2024-03-25T18:15:08.858 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.857+0000 7f8fb57fa640 1 -- 172.21.15.66:0/67681916 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8fac001850 con 0x7f8fb80a98f0 2024-03-25T18:15:08.858 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.857+0000 7f8fb57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:08.858 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.857+0000 7f8fb57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:08.859 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.857+0000 7f8fb57fa640 10 monclient(hunting): dump: 2024-03-25T18:15:08.859 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:15:08.859 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:15:08.859 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:15:08.859 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:15:08.859 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:08.859 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:15:08.859 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:15:08.859 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:15:08.859 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:15:08.859 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:15:08.859 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.857+0000 7f8fb57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:08.861 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.860+0000 7f8fb57fa640 1 -- 172.21.15.66:0/67681916 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660891344 0 0) 0x7f8fb80ac910 con 0x7f8fb81430c0 2024-03-25T18:15:08.861 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.860+0000 7f8fb57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:08.862 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.860+0000 7f8fb57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:08.862 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.860+0000 7f8fb57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:08.862 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.860+0000 7f8fb57fa640 10 monclient(hunting): my global_id is 4732 2024-03-25T18:15:08.862 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.860+0000 7f8fb57fa640 1 -- 172.21.15.66:0/67681916 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f9c004eb0 con 0x7f8fb81430c0 2024-03-25T18:15:08.872 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.870+0000 7f8fb57fa640 1 -- 172.21.15.66:0/67681916 <== mon.0 v1:172.21.15.66:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2288310195 0 0) 0x7f8fb80b2a30 con 0x7f8fb80a98f0 2024-03-25T18:15:08.872 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.870+0000 7f8fb57fa640 1 -- 172.21.15.66:0/67681916 >> v1:172.21.15.66:6790/0 conn(0x7f8fb813f710 legacy=0x7f8fb80ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:08.872 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.870+0000 7f8fb57fa640 1 -- 172.21.15.66:0/67681916 >> v1:172.21.15.193:6789/0 conn(0x7f8fb81430c0 legacy=0x7f8fb81e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:08.872 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.870+0000 7f8fb57fa640 10 monclient: _finish_hunting 0 2024-03-25T18:15:08.872 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.870+0000 7f8fb57fa640 1 monclient: found mon.a 2024-03-25T18:15:08.872 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.870+0000 7f8fb57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:08.872 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.870+0000 7f8fb57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:15:08.872 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.870+0000 7f8fb57fa640 1 -- 172.21.15.66:0/67681916 --> v1:172.21.15.66:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb80abde0 con 0x7f8fb80a98f0 2024-03-25T18:15:08.872 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.870+0000 7f8fb57fa640 10 monclient: _finish_auth 0 2024-03-25T18:15:08.873 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.870+0000 7f8fb57fa640 10 monclient: _check_auth_tickets 2024-03-25T18:15:08.873 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.870+0000 7f8fb57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:08.873 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.870+0000 7f8fbec51640 5 monclient: authenticate success, global_id 4737 2024-03-25T18:15:08.873 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.870+0000 7f8fbec51640 10 monclient: _renew_subs 2024-03-25T18:15:08.873 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.870+0000 7f8fbec51640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:15:08.873 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.870+0000 7f8fbec51640 1 -- 172.21.15.66:0/67681916 --> v1:172.21.15.66:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fb8138f40 con 0x7f8fb80a98f0 2024-03-25T18:15:08.873 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.870+0000 7f8fbec51640 10 client.4737.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:08.873 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.870+0000 7f8fbec51640 10 monclient: _renew_subs 2024-03-25T18:15:08.873 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.870+0000 7f8fbec51640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:15:08.873 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.870+0000 7f8fbec51640 1 -- 172.21.15.66:0/67681916 --> v1:172.21.15.66:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fb81363f0 con 0x7f8fb80a98f0 2024-03-25T18:15:08.873 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.870+0000 7f8fbec51640 1 librados: init done 2024-03-25T18:15:08.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.871+0000 7f8fbec51640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:08.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.872+0000 7f8fbec51640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:08.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.872+0000 7f8fbec51640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:15:08.874 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:08.872+0000 7f8fbec51640 1 -- 172.21.15.66:0/67681916 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fb80627b0 con 0x7f8fb80a98f0 2024-03-25T18:15:09.202 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.201+0000 7f8fb57fa640 1 -- 172.21.15.66:0/67681916 <== mon.0 v1:172.21.15.66:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8fac0047e0 con 0x7f8fb80a98f0 2024-03-25T18:15:09.203 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.201+0000 7f8fb57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:09.203 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.201+0000 7f8fb57fa640 1 -- 172.21.15.66:0/67681916 <== mon.0 v1:172.21.15.66:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f8fac004af0 con 0x7f8fb80a98f0 2024-03-25T18:15:09.203 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.201+0000 7f8fb57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:09.203 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.201+0000 7f8fb57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:09.203 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.201+0000 7f8fb57fa640 10 monclient: dump: 2024-03-25T18:15:09.203 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:15:09.203 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:15:09.203 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:15:09.203 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:15:09.203 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:09.203 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:15:09.203 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:15:09.203 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:15:09.203 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:15:09.203 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:15:09.204 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.201+0000 7f8fb6ffd640 4 set_mon_vals no callback set 2024-03-25T18:15:09.204 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.201+0000 7f8fb57fa640 1 -- 172.21.15.66:0/67681916 <== mon.0 v1:172.21.15.66:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f8fac01aac0 con 0x7f8fb80a98f0 2024-03-25T18:15:09.204 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.202+0000 7f8fb7fff640 10 monclient: get_auth_request con 0x7f8f9c06de20 auth_method 0 2024-03-25T18:15:09.204 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.203+0000 7f8fb57fa640 10 client.4737.objecter ms_handle_connect 0x7f8f9c06de20 2024-03-25T18:15:09.209 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.207+0000 7f8fb57fa640 1 -- 172.21.15.66:0/67681916 <== mon.0 v1:172.21.15.66:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4152745027 0 0) 0x7f8fac0546c0 con 0x7f8fb80a98f0 2024-03-25T18:15:09.209 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.207+0000 7f8fb57fa640 10 client.4737.objecter ms_dispatch 0x7f8fb8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:15:09.209 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.207+0000 7f8fb57fa640 3 client.4737.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:15:09.209 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.207+0000 7f8fb57fa640 3 client.4737.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:15:09.209 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.207+0000 7f8fb57fa640 20 client.4737.objecter dump_active .. 0 homeless 2024-03-25T18:15:09.209 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.207+0000 7f8fb57fa640 1 -- 172.21.15.66:0/67681916 <== mon.0 v1:172.21.15.66:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8fac0501d0 con 0x7f8fb80a98f0 2024-03-25T18:15:09.209 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.207+0000 7f8fb57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:09.209 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.207+0000 7f8fb57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:09.473 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.471+0000 7f8fbec51640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:09.473 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.471+0000 7f8fbec51640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:09.473 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.471+0000 7f8fbec51640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.66:6789/0 2024-03-25T18:15:09.473 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.471+0000 7f8fbec51640 1 -- 172.21.15.66:0/67681916 --> v1:172.21.15.66:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8fb81434a0 con 0x7f8fb80a98f0 2024-03-25T18:15:09.508 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.507+0000 7f8fb57fa640 1 -- 172.21.15.66:0/67681916 <== mon.0 v1:172.21.15.66:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4258808387) 0x7f8fac052100 con 0x7f8fb80a98f0 2024-03-25T18:15:09.508 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.507+0000 7f8fb57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:09.509 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.507+0000 7f8fb57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:09.509 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:15:09.509 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":11,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:13:50.353085+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:50.353085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:09.511 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.509+0000 7f8fbec51640 10 librados: watch_flush enter 2024-03-25T18:15:09.511 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.509+0000 7f8fbec51640 10 librados: watch_flush exit 2024-03-25T18:15:09.511 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.509+0000 7f8fbec51640 20 client.4737.objecter shutdown clearing up homeless session... 2024-03-25T18:15:09.511 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.509+0000 7f8fbec51640 10 client.4737.objecter successfully canceled tick 2024-03-25T18:15:09.511 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.509+0000 7f8fbec51640 1 -- 172.21.15.66:0/67681916 >> v1:172.21.15.193:6800/2275676803 conn(0x7f8f9c06de20 legacy=0x7f8f9c0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:09.511 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.509+0000 7f8fbec51640 10 monclient: shutdown 2024-03-25T18:15:09.511 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.509+0000 7f8fbec51640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:09.512 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.509+0000 7f8fbec51640 1 -- 172.21.15.66:0/67681916 >> v1:172.21.15.66:6789/0 conn(0x7f8fb80a98f0 legacy=0x7f8fb8138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:09.512 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.509+0000 7f8fbec51640 1 -- 172.21.15.66:0/67681916 shutdown_connections 2024-03-25T18:15:09.512 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.509+0000 7f8fbec51640 1 -- 172.21.15.66:0/67681916 >> 172.21.15.66:0/67681916 conn(0x7f8fb80b7ea0 msgr2=0x7f8fb8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:09.512 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.509+0000 7f8fbec51640 1 -- 172.21.15.66:0/67681916 shutdown_connections 2024-03-25T18:15:09.512 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.509+0000 7f8fbec51640 1 -- 172.21.15.66:0/67681916 wait complete. 2024-03-25T18:15:09.512 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:09.509+0000 7f8fbec51640 1 librados: shutdown 2024-03-25T18:15:09.523 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:15.840 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:11:46.820 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:15.846 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:11:46.826 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:15.884 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:11:46.865 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:15.891 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:11:46.849 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:15.896 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:11:46.876 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:15.902 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:11:46.860 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:15.909 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:11:46.890 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:15.934 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:11:46.893 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:15.952 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:11:46.933 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:16.329 DEBUG:teuthology.orchestra.run.smithi066:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:16.450 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.446+0000 7f439e0f1640 10 monclient: get_monmap_and_config 2024-03-25T18:15:16.451 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.446+0000 7f439e0f1640 10 monclient: build_initial_monmap 2024-03-25T18:15:16.451 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.446+0000 7f439e0f1640 10 monclient: monmap: 2024-03-25T18:15:16.451 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:15:16.451 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:15:16.451 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:15:16.447526+0000 2024-03-25T18:15:16.451 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:15:16.447526+0000 2024-03-25T18:15:16.451 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:16.451 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:15:16.451 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:15:16.451 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-c 2024-03-25T18:15:16.451 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-b 2024-03-25T18:15:16.451 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:15:16.451 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.447+0000 7f439e0f1640 1 Processor -- start 2024-03-25T18:15:16.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.447+0000 7f439e0f1640 1 -- start start 2024-03-25T18:15:16.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.447+0000 7f439e0f1640 10 monclient: init 2024-03-25T18:15:16.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.447+0000 7f439e0f1640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:16.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.447+0000 7f439e0f1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:16.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.447+0000 7f439e0f1640 10 monclient(hunting): picked mon.noname-b con 0x7f43980a98f0 addr v1:172.21.15.66:6790/0 2024-03-25T18:15:16.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.447+0000 7f439e0f1640 10 monclient(hunting): picked mon.noname-a con 0x7f439813f710 addr v1:172.21.15.66:6789/0 2024-03-25T18:15:16.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.448+0000 7f439e0f1640 10 monclient(hunting): picked mon.noname-c con 0x7f43981430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:15:16.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.448+0000 7f439e0f1640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4398142520 con 0x7f439813f710 2024-03-25T18:15:16.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.448+0000 7f439e0f1640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43980b28a0 con 0x7f43981430c0 2024-03-25T18:15:16.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.448+0000 7f439e0f1640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43980b2a30 con 0x7f43980a98f0 2024-03-25T18:15:16.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.448+0000 7f439e0f1640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:16.452 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.448+0000 7f439e0f1640 10 monclient(hunting): authenticate will time out at 7665.508232s 2024-03-25T18:15:16.460 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.459+0000 7f43977fe640 1 --1- >> v1:172.21.15.66:6790/0 conn(0x7f43980a98f0 0x7f43980a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.66:6790/0 says I am v1:172.21.15.66:51524/0 (socket says 172.21.15.66:51524) 2024-03-25T18:15:16.460 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.459+0000 7f43977fe640 1 -- 172.21.15.66:0/3048620192 learned_addr learned my addr 172.21.15.66:0/3048620192 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:15:16.470 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.469+0000 7f43967fc640 1 -- 172.21.15.66:0/3048620192 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3045625031 0 0) 0x7f4398142520 con 0x7f439813f710 2024-03-25T18:15:16.470 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.469+0000 7f43967fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:16.470 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.469+0000 7f43967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:16.470 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.469+0000 7f43967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:16.470 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.469+0000 7f43967fc640 10 monclient(hunting): my global_id is 4740 2024-03-25T18:15:16.470 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.469+0000 7f43967fc640 1 -- 172.21.15.66:0/3048620192 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4380003520 con 0x7f439813f710 2024-03-25T18:15:16.476 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.475+0000 7f43967fc640 1 -- 172.21.15.66:0/3048620192 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514769842 0 0) 0x7f43980b28a0 con 0x7f43981430c0 2024-03-25T18:15:16.476 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.475+0000 7f43967fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:16.476 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.475+0000 7f43967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:16.477 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.475+0000 7f43967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:16.477 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.475+0000 7f43967fc640 10 monclient(hunting): my global_id is 4735 2024-03-25T18:15:16.477 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.475+0000 7f43967fc640 1 -- 172.21.15.66:0/3048620192 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4398142520 con 0x7f43981430c0 2024-03-25T18:15:16.486 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.485+0000 7f43967fc640 1 -- 172.21.15.66:0/3048620192 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3625057781 0 0) 0x7f43980b2a30 con 0x7f43980a98f0 2024-03-25T18:15:16.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.485+0000 7f43967fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:16.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.485+0000 7f43967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:16.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.485+0000 7f43967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:16.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.485+0000 7f43967fc640 10 monclient(hunting): my global_id is 4733 2024-03-25T18:15:16.487 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.485+0000 7f43967fc640 1 -- 172.21.15.66:0/3048620192 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43980b28a0 con 0x7f43980a98f0 2024-03-25T18:15:16.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.498+0000 7f43967fc640 1 -- 172.21.15.66:0/3048620192 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 205273182 0 0) 0x7f4398142520 con 0x7f43981430c0 2024-03-25T18:15:16.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.498+0000 7f43967fc640 1 -- 172.21.15.66:0/3048620192 >> v1:172.21.15.66:6790/0 conn(0x7f43980a98f0 legacy=0x7f43980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:16.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.498+0000 7f43967fc640 1 -- 172.21.15.66:0/3048620192 >> v1:172.21.15.66:6789/0 conn(0x7f439813f710 legacy=0x7f43981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:16.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.498+0000 7f43967fc640 10 monclient: _finish_hunting 0 2024-03-25T18:15:16.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.498+0000 7f43967fc640 1 monclient: found mon.noname-c 2024-03-25T18:15:16.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.498+0000 7f43967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:16.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.498+0000 7f43967fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.193:6789/0 2024-03-25T18:15:16.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.498+0000 7f43967fc640 1 -- 172.21.15.66:0/3048620192 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43980b44c0 con 0x7f43981430c0 2024-03-25T18:15:16.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.498+0000 7f43967fc640 10 monclient: _finish_auth 0 2024-03-25T18:15:16.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.498+0000 7f43967fc640 10 monclient: _check_auth_tickets 2024-03-25T18:15:16.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.498+0000 7f43967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:16.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.498+0000 7f43967fc640 1 -- 172.21.15.66:0/3048620192 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f43900019f0 con 0x7f43981430c0 2024-03-25T18:15:16.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.498+0000 7f43967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:16.500 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.498+0000 7f43967fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:16.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.498+0000 7f43967fc640 10 monclient: dump: 2024-03-25T18:15:16.501 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:15:16.501 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:15:16.501 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:15:16.501 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:15:16.501 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:16.501 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:15:16.501 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:15:16.501 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:15:16.501 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:15:16.501 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:15:16.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.498+0000 7f439e0f1640 5 monclient: authenticate success, global_id 4735 2024-03-25T18:15:16.501 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.498+0000 7f439e0f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:16.527 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.526+0000 7f43967fc640 1 -- 172.21.15.66:0/3048620192 <== mon.1 v1:172.21.15.193:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4390001b50 con 0x7f43981430c0 2024-03-25T18:15:16.527 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.526+0000 7f43967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:16.527 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.526+0000 7f43967fc640 1 -- 172.21.15.66:0/3048620192 <== mon.1 v1:172.21.15.193:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f4390001f60 con 0x7f43981430c0 2024-03-25T18:15:16.527 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.526+0000 7f43967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:16.528 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.526+0000 7f43967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:16.528 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.526+0000 7f43967fc640 10 monclient: dump: 2024-03-25T18:15:16.528 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:15:16.528 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:15:16.528 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:15:16.528 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:15:16.528 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:16.528 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:15:16.528 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:15:16.528 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:15:16.528 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:15:16.528 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:15:16.528 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.526+0000 7f439e0f1640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:16.528 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.526+0000 7f439e0f1640 4 set_mon_vals no callback set 2024-03-25T18:15:16.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.527+0000 7f439e0f1640 10 monclient: shutdown 2024-03-25T18:15:16.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.527+0000 7f439e0f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:16.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.527+0000 7f439e0f1640 1 -- 172.21.15.66:0/3048620192 >> v1:172.21.15.193:6789/0 conn(0x7f43981430c0 legacy=0x7f439814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:16.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.527+0000 7f439e0f1640 1 -- 172.21.15.66:0/3048620192 shutdown_connections 2024-03-25T18:15:16.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.527+0000 7f439e0f1640 1 -- 172.21.15.66:0/3048620192 >> 172.21.15.66:0/3048620192 conn(0x7f43980b7ea0 msgr2=0x7f43980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:16.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.527+0000 7f439e0f1640 1 -- 172.21.15.66:0/3048620192 shutdown_connections 2024-03-25T18:15:16.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.527+0000 7f439e0f1640 1 -- 172.21.15.66:0/3048620192 wait complete. 2024-03-25T18:15:16.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.528+0000 7f439e0f1640 10 monclient: build_initial_monmap 2024-03-25T18:15:16.530 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.528+0000 7f439e0f1640 10 monclient: monmap: 2024-03-25T18:15:16.530 INFO:teuthology.misc.health.smithi066.stderr:epoch 0 2024-03-25T18:15:16.530 INFO:teuthology.misc.health.smithi066.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:16.530 INFO:teuthology.misc.health.smithi066.stderr:last_changed 0.000000 2024-03-25T18:15:16.531 INFO:teuthology.misc.health.smithi066.stderr:created 0.000000 2024-03-25T18:15:16.531 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:16.531 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 1 2024-03-25T18:15:16.531 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.noname-a 2024-03-25T18:15:16.531 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.noname-b 2024-03-25T18:15:16.531 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.noname-c 2024-03-25T18:15:16.531 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:15:16.531 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.528+0000 7f439e0f1640 1 librados: starting msgr at 2024-03-25T18:15:16.531 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.528+0000 7f439e0f1640 1 librados: starting objecter 2024-03-25T18:15:16.531 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.528+0000 7f439e0f1640 1 Processor -- start 2024-03-25T18:15:16.532 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.529+0000 7f439e0f1640 1 -- start start 2024-03-25T18:15:16.532 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.529+0000 7f439e0f1640 1 librados: setting wanted keys 2024-03-25T18:15:16.532 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.529+0000 7f439e0f1640 1 librados: calling monclient init 2024-03-25T18:15:16.532 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.529+0000 7f439e0f1640 10 monclient: init 2024-03-25T18:15:16.532 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.529+0000 7f439e0f1640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:16.532 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.529+0000 7f439e0f1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:15:16.533 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.529+0000 7f439e0f1640 10 monclient(hunting): picked mon.noname-a con 0x7f43980a98f0 addr v1:172.21.15.66:6789/0 2024-03-25T18:15:16.533 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.530+0000 7f439e0f1640 10 monclient(hunting): picked mon.noname-c con 0x7f439813f710 addr v1:172.21.15.66:6790/0 2024-03-25T18:15:16.533 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.530+0000 7f439e0f1640 10 monclient(hunting): picked mon.noname-b con 0x7f43981430c0 addr v1:172.21.15.193:6789/0 2024-03-25T18:15:16.533 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.530+0000 7f439e0f1640 1 -- --> v1:172.21.15.66:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43981388a0 con 0x7f43980a98f0 2024-03-25T18:15:16.533 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.530+0000 7f439e0f1640 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43981349e0 con 0x7f43981430c0 2024-03-25T18:15:16.533 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.530+0000 7f439e0f1640 1 -- --> v1:172.21.15.66:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4398138d30 con 0x7f439813f710 2024-03-25T18:15:16.533 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.530+0000 7f439e0f1640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:16.533 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.530+0000 7f439e0f1640 10 monclient(hunting): authenticate will time out at 7665.590236s 2024-03-25T18:15:16.537 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.535+0000 7f43977fe640 1 --1- >> v1:172.21.15.66:6789/0 conn(0x7f43980a98f0 0x7f43981da880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.66:6789/0 says I am v1:172.21.15.66:59552/0 (socket says 172.21.15.66:59552) 2024-03-25T18:15:16.537 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.535+0000 7f43977fe640 1 -- 172.21.15.66:0/1329911612 learned_addr learned my addr 172.21.15.66:0/1329911612 (peer_addr_for_me v1:172.21.15.66:0/0) 2024-03-25T18:15:16.539 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.537+0000 7f437ffff640 10 client.?.objecter ms_handle_connect 0x7f43980a98f0 2024-03-25T18:15:16.539 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.537+0000 7f437ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:16.544 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.543+0000 7f437ffff640 1 -- 172.21.15.66:0/1329911612 <== mon.0 v1:172.21.15.66:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3107181920 0 0) 0x7f43981388a0 con 0x7f43980a98f0 2024-03-25T18:15:16.544 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.543+0000 7f437ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:16.544 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.543+0000 7f437ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:16.545 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.543+0000 7f437ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:16.545 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.543+0000 7f437ffff640 10 monclient(hunting): my global_id is 4743 2024-03-25T18:15:16.545 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.543+0000 7f437ffff640 1 -- 172.21.15.66:0/1329911612 --> v1:172.21.15.66:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4384004eb0 con 0x7f43980a98f0 2024-03-25T18:15:16.545 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.543+0000 7f437ffff640 10 client.?.objecter ms_handle_connect 0x7f43981430c0 2024-03-25T18:15:16.545 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.543+0000 7f437ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:16.547 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.546+0000 7f437ffff640 10 client.?.objecter ms_handle_connect 0x7f439813f710 2024-03-25T18:15:16.547 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.546+0000 7f437ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:16.556 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.555+0000 7f437ffff640 1 -- 172.21.15.66:0/1329911612 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2326892341 0 0) 0x7f43981349e0 con 0x7f43981430c0 2024-03-25T18:15:16.556 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.555+0000 7f437ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:16.556 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.555+0000 7f437ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:16.556 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.555+0000 7f437ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:16.556 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.555+0000 7f437ffff640 10 monclient(hunting): my global_id is 4738 2024-03-25T18:15:16.557 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.555+0000 7f437ffff640 1 -- 172.21.15.66:0/1329911612 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43981388a0 con 0x7f43981430c0 2024-03-25T18:15:16.562 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.560+0000 7f437ffff640 1 -- 172.21.15.66:0/1329911612 <== mon.0 v1:172.21.15.66:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3730498389 0 0) 0x7f4384004eb0 con 0x7f43980a98f0 2024-03-25T18:15:16.562 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.560+0000 7f437ffff640 1 -- 172.21.15.66:0/1329911612 --> v1:172.21.15.66:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43981349e0 con 0x7f43980a98f0 2024-03-25T18:15:16.562 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.560+0000 7f437ffff640 1 -- 172.21.15.66:0/1329911612 <== mon.0 v1:172.21.15.66:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f4388001910 con 0x7f43980a98f0 2024-03-25T18:15:16.562 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.560+0000 7f437ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:16.562 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.560+0000 7f437ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:16.563 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.560+0000 7f437ffff640 10 monclient(hunting): dump: 2024-03-25T18:15:16.563 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:15:16.563 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:15:16.563 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:15:16.563 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:15:16.563 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:16.563 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:15:16.563 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:15:16.563 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:15:16.563 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:15:16.563 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:15:16.563 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.560+0000 7f437ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:16.563 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.561+0000 7f437ffff640 1 -- 172.21.15.66:0/1329911612 <== mon.2 v1:172.21.15.66:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3555505986 0 0) 0x7f4398138d30 con 0x7f439813f710 2024-03-25T18:15:16.563 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.561+0000 7f437ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:16.563 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.561+0000 7f437ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:16.563 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.561+0000 7f437ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:16.563 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.561+0000 7f437ffff640 10 monclient(hunting): my global_id is 4736 2024-03-25T18:15:16.564 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.561+0000 7f437ffff640 1 -- 172.21.15.66:0/1329911612 --> v1:172.21.15.66:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4384004eb0 con 0x7f439813f710 2024-03-25T18:15:16.568 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.567+0000 7f437ffff640 1 -- 172.21.15.66:0/1329911612 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3666714917 0 0) 0x7f43981388a0 con 0x7f43981430c0 2024-03-25T18:15:16.568 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.567+0000 7f437ffff640 1 -- 172.21.15.66:0/1329911612 --> v1:172.21.15.193:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4398138d30 con 0x7f43981430c0 2024-03-25T18:15:16.568 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.567+0000 7f437ffff640 1 -- 172.21.15.66:0/1329911612 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f4390001e60 con 0x7f43981430c0 2024-03-25T18:15:16.569 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.567+0000 7f437ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:16.569 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.567+0000 7f437ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:16.569 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.567+0000 7f437ffff640 10 monclient(hunting): dump: 2024-03-25T18:15:16.569 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:15:16.569 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:15:16.569 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:15:16.569 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:15:16.569 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:16.569 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:15:16.569 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:15:16.569 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:15:16.569 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:15:16.569 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:15:16.578 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.576+0000 7f437ffff640 1 -- 172.21.15.66:0/1329911612 <== mon.1 v1:172.21.15.193:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3827656112 0 0) 0x7f4398138d30 con 0x7f43981430c0 2024-03-25T18:15:16.578 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.576+0000 7f437ffff640 1 -- 172.21.15.66:0/1329911612 >> v1:172.21.15.66:6790/0 conn(0x7f439813f710 legacy=0x7f4398133bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:16.578 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.576+0000 7f437ffff640 1 -- 172.21.15.66:0/1329911612 >> v1:172.21.15.66:6789/0 conn(0x7f43980a98f0 legacy=0x7f43981da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:16.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.576+0000 7f437ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:15:16.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.576+0000 7f437ffff640 1 monclient: found mon.b 2024-03-25T18:15:16.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.576+0000 7f437ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:16.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.576+0000 7f437ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:15:16.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.576+0000 7f437ffff640 1 -- 172.21.15.66:0/1329911612 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43980aba50 con 0x7f43981430c0 2024-03-25T18:15:16.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.576+0000 7f437ffff640 10 monclient: _finish_auth 0 2024-03-25T18:15:16.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.576+0000 7f437ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:15:16.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.576+0000 7f437ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:16.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.576+0000 7f439e0f1640 5 monclient: authenticate success, global_id 4738 2024-03-25T18:15:16.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.576+0000 7f439e0f1640 10 monclient: _renew_subs 2024-03-25T18:15:16.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.576+0000 7f439e0f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:15:16.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.576+0000 7f439e0f1640 1 -- 172.21.15.66:0/1329911612 --> v1:172.21.15.193:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43980b1770 con 0x7f43981430c0 2024-03-25T18:15:16.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.576+0000 7f439e0f1640 10 client.4738.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:16.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.576+0000 7f439e0f1640 10 monclient: _renew_subs 2024-03-25T18:15:16.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.576+0000 7f439e0f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:15:16.579 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.576+0000 7f439e0f1640 1 -- 172.21.15.66:0/1329911612 --> v1:172.21.15.193:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4398137ba0 con 0x7f43981430c0 2024-03-25T18:15:16.580 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.576+0000 7f439e0f1640 1 librados: init done 2024-03-25T18:15:16.580 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.577+0000 7f439e0f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:16.580 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.577+0000 7f439e0f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:16.580 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.577+0000 7f439e0f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:15:16.580 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.577+0000 7f439e0f1640 1 -- 172.21.15.66:0/1329911612 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43980627b0 con 0x7f43981430c0 2024-03-25T18:15:16.849 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.848+0000 7f437ffff640 1 -- 172.21.15.66:0/1329911612 <== mon.1 v1:172.21.15.193:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4390002020 con 0x7f43981430c0 2024-03-25T18:15:16.849 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.848+0000 7f437ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:16.849 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.848+0000 7f437ffff640 1 -- 172.21.15.66:0/1329911612 <== mon.1 v1:172.21.15.193:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3000164040 0 0) 0x7f4390003dc0 con 0x7f43981430c0 2024-03-25T18:15:16.849 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.848+0000 7f437ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:16.850 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.848+0000 7f437ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:16.850 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.848+0000 7f437ffff640 10 monclient: dump: 2024-03-25T18:15:16.850 INFO:teuthology.misc.health.smithi066.stderr:epoch 1 2024-03-25T18:15:16.850 INFO:teuthology.misc.health.smithi066.stderr:fsid 8da823bc-9cd7-40a4-b5f1-07036a791c17 2024-03-25T18:15:16.850 INFO:teuthology.misc.health.smithi066.stderr:last_changed 2024-03-25T18:02:55.929266+0000 2024-03-25T18:15:16.850 INFO:teuthology.misc.health.smithi066.stderr:created 2024-03-25T18:02:55.929266+0000 2024-03-25T18:15:16.850 INFO:teuthology.misc.health.smithi066.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:16.850 INFO:teuthology.misc.health.smithi066.stderr:election_strategy: 3 2024-03-25T18:15:16.850 INFO:teuthology.misc.health.smithi066.stderr:0: v1:172.21.15.66:6789/0 mon.a 2024-03-25T18:15:16.850 INFO:teuthology.misc.health.smithi066.stderr:1: v1:172.21.15.193:6789/0 mon.b 2024-03-25T18:15:16.850 INFO:teuthology.misc.health.smithi066.stderr:2: v1:172.21.15.66:6790/0 mon.c 2024-03-25T18:15:16.850 INFO:teuthology.misc.health.smithi066.stderr: 2024-03-25T18:15:16.850 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.848+0000 7f4395ffb640 4 set_mon_vals no callback set 2024-03-25T18:15:16.851 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.849+0000 7f437ffff640 1 -- 172.21.15.66:0/1329911612 <== mon.1 v1:172.21.15.193:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2821939513 0 0) 0x7f4390019d90 con 0x7f43981430c0 2024-03-25T18:15:16.851 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.849+0000 7f43977fe640 10 monclient: get_auth_request con 0x7f438406dec0 auth_method 0 2024-03-25T18:15:16.851 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.849+0000 7f437ffff640 10 client.4738.objecter ms_handle_connect 0x7f438406dec0 2024-03-25T18:15:16.855 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.853+0000 7f437ffff640 1 -- 172.21.15.66:0/1329911612 <== mon.1 v1:172.21.15.193:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4152745027 0 0) 0x7f4390053970 con 0x7f43981430c0 2024-03-25T18:15:16.855 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.853+0000 7f437ffff640 10 client.4738.objecter ms_dispatch 0x7f4398000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:15:16.855 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.853+0000 7f437ffff640 3 client.4738.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:15:16.855 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.853+0000 7f437ffff640 3 client.4738.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:15:16.855 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.853+0000 7f437ffff640 20 client.4738.objecter dump_active .. 0 homeless 2024-03-25T18:15:16.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.855+0000 7f437ffff640 1 -- 172.21.15.66:0/1329911612 <== mon.1 v1:172.21.15.193:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f439004f2b0 con 0x7f43981430c0 2024-03-25T18:15:16.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.855+0000 7f437ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:16.856 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:16.855+0000 7f437ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:17.075 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:17.073+0000 7f439e0f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:17.075 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:17.073+0000 7f439e0f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:17.075 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:17.073+0000 7f439e0f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.193:6789/0 2024-03-25T18:15:17.075 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:17.073+0000 7f439e0f1640 1 -- 172.21.15.66:0/1329911612 --> v1:172.21.15.193:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4398138a50 con 0x7f43981430c0 2024-03-25T18:15:17.148 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:17.147+0000 7f437ffff640 1 -- 172.21.15.66:0/1329911612 <== mon.1 v1:172.21.15.193:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 4258808387) 0x7f4398138a50 con 0x7f43981430c0 2024-03-25T18:15:17.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:17.147+0000 7f437ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:17.149 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:17.147+0000 7f437ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:17.149 INFO:teuthology.misc.health.smithi066.stdout: 2024-03-25T18:15:17.149 INFO:teuthology.misc.health.smithi066.stdout:{"epoch":11,"fsid":"8da823bc-9cd7-40a4-b5f1-07036a791c17","created":"2024-03-25T18:03:46.430566+0000","modified":"2024-03-25T18:13:50.353085+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:50.353085+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"508b038c-3081-4abb-aaa7-67b50c558552","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6335b90f-0088-47db-932c-e693dff69998","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"d618ea26-b83d-46e1-bd9c-439c7d6dacc4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"b3c5bd8b-e425-40d6-9208-aa973ce3c777","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"34c9396f-b15a-43ed-aef7-ef55aa31d14d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0ae5db93-018c-48ff-a423-7061d004c4c1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c6dbdea7-0bf1-43ca-8bbc-ce8fc60f72dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"59a326e8-ae37-42fc-a22c-24b55a726954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:17.152 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:17.149+0000 7f439e0f1640 10 librados: watch_flush enter 2024-03-25T18:15:17.152 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:17.150+0000 7f439e0f1640 10 librados: watch_flush exit 2024-03-25T18:15:17.152 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:17.150+0000 7f439e0f1640 20 client.4738.objecter shutdown clearing up homeless session... 2024-03-25T18:15:17.152 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:17.150+0000 7f439e0f1640 10 client.4738.objecter successfully canceled tick 2024-03-25T18:15:17.152 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:17.150+0000 7f439e0f1640 1 -- 172.21.15.66:0/1329911612 >> v1:172.21.15.193:6800/2275676803 conn(0x7f438406dec0 legacy=0x7f4384070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:17.152 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:17.150+0000 7f439e0f1640 10 monclient: shutdown 2024-03-25T18:15:17.152 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:17.150+0000 7f439e0f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:17.152 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:17.150+0000 7f439e0f1640 1 -- 172.21.15.66:0/1329911612 >> v1:172.21.15.193:6789/0 conn(0x7f43981430c0 legacy=0x7f43981342d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:17.152 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:17.150+0000 7f439e0f1640 1 -- 172.21.15.66:0/1329911612 shutdown_connections 2024-03-25T18:15:17.152 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:17.150+0000 7f439e0f1640 1 -- 172.21.15.66:0/1329911612 >> 172.21.15.66:0/1329911612 conn(0x7f43980b7ea0 msgr2=0x7f4398074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:17.152 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:17.150+0000 7f439e0f1640 1 -- 172.21.15.66:0/1329911612 shutdown_connections 2024-03-25T18:15:17.152 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:17.150+0000 7f439e0f1640 1 -- 172.21.15.66:0/1329911612 wait complete. 2024-03-25T18:15:17.152 INFO:teuthology.misc.health.smithi066.stderr:2024-03-25T18:15:17.150+0000 7f439e0f1640 1 librados: shutdown 2024-03-25T18:15:17.163 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:17.163 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T18:15:17.165 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T18:15:17.165 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T18:15:17.165 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:15:19.125 INFO:tasks.ceph.osd.0.smithi066.stderr:==00:00:11:26.496 35448== 2024-03-25T18:15:19.125 INFO:tasks.ceph.osd.0.smithi066.stderr:==00:00:11:26.496 35448== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:15:19.162 INFO:tasks.ceph.osd.0.smithi066.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:15:23.267 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:15:23.268 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi066 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T18:15:23.268 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T18:15:23.269 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T18:15:23.269 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:15:25.158 INFO:tasks.ceph.osd.1.smithi066.stderr:==00:00:11:32.533 35447== 2024-03-25T18:15:25.159 INFO:tasks.ceph.osd.1.smithi066.stderr:==00:00:11:32.533 35447== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:15:25.197 INFO:tasks.ceph.osd.1.smithi066.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:15:29.371 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:15:29.371 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi066 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T18:15:29.371 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T18:15:29.372 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T18:15:29.372 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:15:31.457 INFO:tasks.ceph.osd.2.smithi066.stderr:==00:00:11:38.825 35449== 2024-03-25T18:15:31.457 INFO:tasks.ceph.osd.2.smithi066.stderr:==00:00:11:38.825 35449== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:15:31.497 INFO:tasks.ceph.osd.2.smithi066.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:15:35.473 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:15:35.474 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi066 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T18:15:35.474 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T18:15:35.474 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T18:15:35.474 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:15:35.837 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:12:06.817 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:35.911 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:12:06.891 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:35.932 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:12:06.912 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:35.951 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:12:06.909 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:35.957 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:12:06.915 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:35.977 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:12:06.936 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:35.979 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:12:06.960 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:35.985 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:12:06.966 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:36.012 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:12:06.993 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:37.254 INFO:tasks.ceph.osd.3.smithi066.stderr:==00:00:11:44.615 35450== 2024-03-25T18:15:37.254 INFO:tasks.ceph.osd.3.smithi066.stderr:==00:00:11:44.615 35450== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:15:37.292 INFO:tasks.ceph.osd.3.smithi066.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:15:41.576 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:15:41.576 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi066 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T18:15:41.577 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T18:15:41.577 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T18:15:41.577 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:15:43.630 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:11:51.020 35227== 2024-03-25T18:15:43.631 INFO:tasks.ceph.osd.4.smithi193.stderr:==00:00:11:51.020 35227== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:15:43.667 INFO:tasks.ceph.osd.4.smithi193.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:15:47.679 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:15:47.679 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi193 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T18:15:47.680 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T18:15:47.680 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T18:15:47.680 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:15:49.643 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:11:57.028 35228== 2024-03-25T18:15:49.643 INFO:tasks.ceph.osd.5.smithi193.stderr:==00:00:11:57.028 35228== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:15:49.682 INFO:tasks.ceph.osd.5.smithi193.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:15:53.782 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:15:53.782 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi193 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T18:15:53.783 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T18:15:53.783 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T18:15:53.783 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:15:55.502 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:12:02.887 35229== 2024-03-25T18:15:55.502 INFO:tasks.ceph.osd.6.smithi193.stderr:==00:00:12:02.887 35229== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:15:55.541 INFO:tasks.ceph.osd.6.smithi193.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:15:55.884 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:12:26.865 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:55.935 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:12:26.916 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:55.949 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:12:26.929 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:55.978 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:12:26.936 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:55.979 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:12:26.937 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:55.980 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:12:26.961 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:55.982 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:12:26.963 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:55.997 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:12:26.955 34648== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:56.000 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:12:26.980 35140== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:59.885 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:15:59.885 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi193 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T18:15:59.885 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T18:15:59.886 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T18:15:59.886 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:16:01.307 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:12:08.686 35230== 2024-03-25T18:16:01.307 INFO:tasks.ceph.osd.7.smithi193.stderr:==00:00:12:08.686 35230== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:16:01.336 INFO:tasks.ceph.osd.7.smithi193.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:16:05.987 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:16:05.988 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi193 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T18:16:05.988 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T18:16:05.988 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T18:16:05.989 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T18:16:05.989 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:16:06.016 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T18:16:06.016 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T18:16:06.017 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:16:06.018 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T18:16:06.017+0000 7f5e05f91640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 34646) UID: 0 2024-03-25T18:16:06.018 INFO:tasks.ceph.mgr.x.smithi193.stderr:2024-03-25T18:16:06.017+0000 7f5e05f91640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T18:16:06.047 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T18:16:06.048 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T18:16:06.048 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T18:16:06.048 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:16:06.073 INFO:tasks.ceph.mon.a.smithi066.stderr:2024-03-25T18:16:06.061+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35102) UID: 0 2024-03-25T18:16:06.073 INFO:tasks.ceph.mon.a.smithi066.stderr:2024-03-25T18:16:06.062+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T18:16:06.668 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:12:37.649 35138== 2024-03-25T18:16:06.669 INFO:tasks.ceph.mon.a.smithi066.stderr:==00:00:12:37.649 35138== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:16:06.678 INFO:tasks.ceph.mon.a.smithi066.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:16:12.149 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:16:12.149 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi066 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T18:16:12.150 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T18:16:12.150 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T18:16:12.150 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:16:12.167 INFO:tasks.ceph.mon.c.smithi066.stderr:2024-03-25T18:16:12.154+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35104) UID: 0 2024-03-25T18:16:12.167 INFO:tasks.ceph.mon.c.smithi066.stderr:2024-03-25T18:16:12.156+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T18:16:12.846 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:12:43.826 35140== 2024-03-25T18:16:12.846 INFO:tasks.ceph.mon.c.smithi066.stderr:==00:00:12:43.826 35140== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:16:12.855 INFO:tasks.ceph.mon.c.smithi066.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:16:18.252 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:16:18.252 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi066 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T18:16:18.252 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T18:16:18.253 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T18:16:18.253 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:16:18.268 INFO:tasks.ceph.mon.b.smithi193.stderr:2024-03-25T18:16:18.256+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34644) UID: 0 2024-03-25T18:16:18.269 INFO:tasks.ceph.mon.b.smithi193.stderr:2024-03-25T18:16:18.257+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T18:16:18.717 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:12:49.676 34648== 2024-03-25T18:16:18.718 INFO:tasks.ceph.mon.b.smithi193.stderr:==00:00:12:49.676 34648== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:16:18.727 INFO:tasks.ceph.mon.b.smithi193.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:16:24.354 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:16:24.355 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi193 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T18:16:24.355 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T18:16:24.355 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T18:16:24.356 DEBUG:teuthology.orchestra.run.smithi066:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T18:16:24.405 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi066.front.sepia.ceph.com 2024-03-25T18:16:24.405 DEBUG:teuthology.orchestra.run.smithi066:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T18:16:24.549 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi066.front.sepia.ceph.com 2024-03-25T18:16:24.550 DEBUG:teuthology.orchestra.run.smithi066:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T18:16:24.606 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi066.front.sepia.ceph.com 2024-03-25T18:16:24.607 DEBUG:teuthology.orchestra.run.smithi066:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T18:16:24.665 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi066.front.sepia.ceph.com 2024-03-25T18:16:24.666 DEBUG:teuthology.orchestra.run.smithi066:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T18:16:24.771 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi193.front.sepia.ceph.com 2024-03-25T18:16:24.772 DEBUG:teuthology.orchestra.run.smithi193:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T18:16:24.927 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi193.front.sepia.ceph.com 2024-03-25T18:16:24.927 DEBUG:teuthology.orchestra.run.smithi193:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T18:16:24.995 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi193.front.sepia.ceph.com 2024-03-25T18:16:24.995 DEBUG:teuthology.orchestra.run.smithi193:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T18:16:25.067 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi193.front.sepia.ceph.com 2024-03-25T18:16:25.067 DEBUG:teuthology.orchestra.run.smithi193:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T18:16:25.132 INFO:tasks.ceph:Archiving mon data... 2024-03-25T18:16:25.135 DEBUG:teuthology.misc:Transferring archived files from smithi066:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621249/data/mon.a.tgz 2024-03-25T18:16:25.135 DEBUG:teuthology.orchestra.run.smithi066:> mktemp 2024-03-25T18:16:25.151 INFO:teuthology.orchestra.run.smithi066.stdout:/tmp/tmp.O2gj6zFM7S 2024-03-25T18:16:25.151 DEBUG:teuthology.orchestra.run.smithi066:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.O2gj6zFM7S 2024-03-25T18:16:25.275 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 0666 /tmp/tmp.O2gj6zFM7S 2024-03-25T18:16:25.364 DEBUG:teuthology.orchestra.remote:smithi066:/tmp/tmp.O2gj6zFM7S is 134KB 2024-03-25T18:16:25.416 DEBUG:teuthology.orchestra.run.smithi066:> rm -fr /tmp/tmp.O2gj6zFM7S 2024-03-25T18:16:25.431 DEBUG:teuthology.misc:Transferring archived files from smithi066:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621249/data/mon.c.tgz 2024-03-25T18:16:25.431 DEBUG:teuthology.orchestra.run.smithi066:> mktemp 2024-03-25T18:16:25.485 INFO:teuthology.orchestra.run.smithi066.stdout:/tmp/tmp.wvRstW0OPH 2024-03-25T18:16:25.485 DEBUG:teuthology.orchestra.run.smithi066:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.wvRstW0OPH 2024-03-25T18:16:25.577 DEBUG:teuthology.orchestra.run.smithi066:> sudo chmod 0666 /tmp/tmp.wvRstW0OPH 2024-03-25T18:16:25.665 DEBUG:teuthology.orchestra.remote:smithi066:/tmp/tmp.wvRstW0OPH is 134KB 2024-03-25T18:16:25.717 DEBUG:teuthology.orchestra.run.smithi066:> rm -fr /tmp/tmp.wvRstW0OPH 2024-03-25T18:16:25.731 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621249/data/mon.b.tgz 2024-03-25T18:16:25.731 DEBUG:teuthology.orchestra.run.smithi193:> mktemp 2024-03-25T18:16:25.749 INFO:teuthology.orchestra.run.smithi193.stdout:/tmp/tmp.yHAQc55U2A 2024-03-25T18:16:25.749 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.yHAQc55U2A 2024-03-25T18:16:25.850 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 0666 /tmp/tmp.yHAQc55U2A 2024-03-25T18:16:25.937 DEBUG:teuthology.orchestra.remote:smithi193:/tmp/tmp.yHAQc55U2A is 134KB 2024-03-25T18:16:25.988 DEBUG:teuthology.orchestra.run.smithi193:> rm -fr /tmp/tmp.yHAQc55U2A 2024-03-25T18:16:26.003 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T18:16:26.003 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:16:26.006 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:16:26.312 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:16:26.554 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T18:16:26.556 DEBUG:teuthology.misc:Transferring archived files from smithi066:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621249/remote/smithi066/crash 2024-03-25T18:16:26.557 DEBUG:teuthology.orchestra.run.smithi066:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:16:26.588 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621249/remote/smithi193/crash 2024-03-25T18:16:26.589 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:16:26.617 INFO:tasks.ceph:Compressing logs... 2024-03-25T18:16:26.617 DEBUG:teuthology.orchestra.run.smithi066:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:16:26.633 DEBUG:teuthology.orchestra.run.smithi193:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:16:26.660 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40029.log 2024-03-25T18:16:26.661 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40850.log 2024-03-25T18:16:26.661 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.40029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41046.log 2024-03-25T18:16:26.661 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.40850.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T18:16:26.661 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39873.log 2024-03-25T18:16:26.700 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T18:16:26.700 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35094.log 2024-03-25T18:16:26.700 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T18:16:26.700 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-client.admin.35094.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T18:16:26.700 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T18:16:26.700 INFO:teuthology.orchestra.run.smithi193.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.35094.log.gz 2024-03-25T18:16:26.700 INFO:teuthology.orchestra.run.smithi193.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T18:16:26.700 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T18:16:26.700 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/valgrind/osd.5.log: /var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T18:16:26.700 INFO:teuthology.orchestra.run.smithi193.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T18:16:26.701 INFO:teuthology.orchestra.run.smithi193.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T18:16:26.701 INFO:teuthology.orchestra.run.smithi193.stderr:gzip/var/log/ceph/valgrind/mon.b.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34923.log 2024-03-25T18:16:26.701 INFO:teuthology.orchestra.run.smithi193.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T18:16:26.701 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34803.log 2024-03-25T18:16:26.701 INFO:teuthology.orchestra.run.smithi193.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T18:16:26.701 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-client.admin.34923.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:16:26.701 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-client.admin.34803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34843.log 2024-03-25T18:16:26.701 INFO:teuthology.orchestra.run.smithi193.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34923.log.gz 2024-03-25T18:16:26.701 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T18:16:26.701 INFO:teuthology.orchestra.run.smithi193.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.34803.log.gz 2024-03-25T18:16:26.701 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-client.admin.34843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35005.log 2024-03-25T18:16:26.701 INFO:teuthology.orchestra.run.smithi193.stderr: 88.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:16:26.701 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-osd.4.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34843.log.gz 2024-03-25T18:16:26.701 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34963.log 2024-03-25T18:16:26.701 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T18:16:26.701 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-client.admin.35005.log: /var/log/ceph/ceph-client.admin.34963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35054.log 2024-03-25T18:16:26.701 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-osd.7.log: gzip 87.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.35005.log.gz --verbose 2024-03-25T18:16:26.702 INFO:teuthology.orchestra.run.smithi193.stderr: -- /var/log/ceph/ceph-osd.5.log 2024-03-25T18:16:26.702 INFO:teuthology.orchestra.run.smithi193.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34963.log.gz 2024-03-25T18:16:26.702 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-client.admin.35054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34883.log 2024-03-25T18:16:26.702 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T18:16:26.702 INFO:teuthology.orchestra.run.smithi193.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35054.log.gz 2024-03-25T18:16:26.702 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-client.admin.34883.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:16:26.702 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34883.log.gz 2024-03-25T18:16:26.702 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:16:26.702 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:16:26.707 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40811.log 2024-03-25T18:16:26.707 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39210.log 2024-03-25T18:16:26.707 INFO:teuthology.orchestra.run.smithi066.stderr: 86.5%/var/log/ceph/ceph-client.admin.41046.log: -- replaced with /var/log/ceph/ceph-client.admin.40029.log.gz 2024-03-25T18:16:26.707 INFO:teuthology.orchestra.run.smithi066.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40850.log.gz 2024-03-25T18:16:26.707 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.39873.log: /var/log/ceph/ceph-client.admin.40811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37588.log 2024-03-25T18:16:26.707 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.39210.log: gzip -5 --verbose -- 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41046.log.gz 2024-03-25T18:16:26.707 INFO:teuthology.orchestra.run.smithi066.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40811.log.gz 2024-03-25T18:16:26.708 INFO:teuthology.orchestra.run.smithi066.stderr: /var/log/ceph/ceph-client.admin.40224.log 2024-03-25T18:16:26.708 INFO:teuthology.orchestra.run.smithi066.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39873.log.gz 2024-03-25T18:16:26.708 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.37588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41397.log 2024-03-25T18:16:26.708 INFO:teuthology.orchestra.run.smithi066.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39210.log.gz 2024-03-25T18:16:26.708 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.40224.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.37588.log.gz 2024-03-25T18:16:26.708 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36504.log 2024-03-25T18:16:26.708 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.41397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40889.log 2024-03-25T18:16:26.708 INFO:teuthology.orchestra.run.smithi066.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40224.log.gz 2024-03-25T18:16:26.708 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.36504.log: gzip 86.7% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.41397.log.gz -- 2024-03-25T18:16:26.708 INFO:teuthology.orchestra.run.smithi066.stderr: /var/log/ceph/ceph-client.admin.41904.log 2024-03-25T18:16:26.708 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.40889.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T18:16:26.708 INFO:teuthology.orchestra.run.smithi066.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36504.log.gz 2024-03-25T18:16:26.708 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.41904.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T18:16:26.708 INFO:teuthology.orchestra.run.smithi066.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40889.log.gz 2024-03-25T18:16:26.708 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T18:16:26.708 INFO:teuthology.orchestra.run.smithi066.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41904.log.gz 2024-03-25T18:16:26.709 INFO:teuthology.orchestra.run.smithi066.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T18:16:26.709 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T18:16:26.709 INFO:teuthology.orchestra.run.smithi066.stderr: 71.3% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T18:16:26.709 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T18:16:26.709 INFO:teuthology.orchestra.run.smithi066.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T18:16:26.709 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T18:16:26.709 INFO:teuthology.orchestra.run.smithi066.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T18:16:26.709 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41202.log 2024-03-25T18:16:26.709 INFO:teuthology.orchestra.run.smithi066.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T18:16:26.709 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40107.log 2024-03-25T18:16:26.709 INFO:teuthology.orchestra.run.smithi066.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T18:16:26.709 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.41202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40068.log 2024-03-25T18:16:26.709 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.40107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41007.log 2024-03-25T18:16:26.709 INFO:teuthology.orchestra.run.smithi066.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41202.log.gz 2024-03-25T18:16:26.709 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.40068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39288.log 2024-03-25T18:16:26.709 INFO:teuthology.orchestra.run.smithi066.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40107.log.gz 2024-03-25T18:16:26.710 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40263.log 2024-03-25T18:16:26.710 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.41007.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40068.log.gz 2024-03-25T18:16:26.710 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.39288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40692.log 2024-03-25T18:16:26.710 INFO:teuthology.orchestra.run.smithi066.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41007.log.gz 2024-03-25T18:16:26.710 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.40263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39561.log 2024-03-25T18:16:26.710 INFO:teuthology.orchestra.run.smithi066.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39288.log.gz 2024-03-25T18:16:26.710 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.40692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40536.log 2024-03-25T18:16:26.710 INFO:teuthology.orchestra.run.smithi066.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40263.log.gz 2024-03-25T18:16:26.710 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.39561.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40692.log.gz 2024-03-25T18:16:26.710 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39366.log 2024-03-25T18:16:26.710 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.40536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40929.log 2024-03-25T18:16:26.710 INFO:teuthology.orchestra.run.smithi066.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39561.log.gz 2024-03-25T18:16:26.710 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.39366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41358.log 2024-03-25T18:16:26.710 INFO:teuthology.orchestra.run.smithi066.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40536.log.gz 2024-03-25T18:16:26.710 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.40929.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39366.log.gz 2024-03-25T18:16:26.710 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41553.log 2024-03-25T18:16:26.711 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.41358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39249.log 2024-03-25T18:16:26.711 INFO:teuthology.orchestra.run.smithi066.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40929.log.gz 2024-03-25T18:16:26.711 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.41553.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:16:26.711 INFO:teuthology.orchestra.run.smithi066.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41358.log.gz 2024-03-25T18:16:26.711 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.39249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41475.log 2024-03-25T18:16:26.711 INFO:teuthology.orchestra.run.smithi066.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41553.log.gz 2024-03-25T18:16:26.711 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39834.log 2024-03-25T18:16:26.711 INFO:teuthology.orchestra.run.smithi066.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39249.log.gz 2024-03-25T18:16:26.711 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.41475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42099.log 2024-03-25T18:16:26.711 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.39834.log: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:16:26.711 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42021.log 2024-03-25T18:16:26.711 INFO:teuthology.orchestra.run.smithi066.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41475.log.gz 2024-03-25T18:16:26.711 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38973.log 2024-03-25T18:16:26.711 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.42099.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39834.log.gz 2024-03-25T18:16:26.711 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.42021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39600.log 2024-03-25T18:16:26.711 INFO:teuthology.orchestra.run.smithi066.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42099.log.gz 2024-03-25T18:16:26.711 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.38973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39522.log 2024-03-25T18:16:26.712 INFO:teuthology.orchestra.run.smithi066.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42021.log.gz 2024-03-25T18:16:26.712 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.39600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39717.log 2024-03-25T18:16:26.712 INFO:teuthology.orchestra.run.smithi066.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38973.log.gz 2024-03-25T18:16:26.712 INFO:teuthology.orchestra.run.smithi066.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39600.log.gz 2024-03-25T18:16:26.712 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.39522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35473.log 2024-03-25T18:16:26.712 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.39717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42060.log 2024-03-25T18:16:26.712 INFO:teuthology.orchestra.run.smithi066.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39522.log.gz 2024-03-25T18:16:26.712 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.35473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40380.log 2024-03-25T18:16:26.712 INFO:teuthology.orchestra.run.smithi066.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39717.log.gz 2024-03-25T18:16:26.712 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.42060.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.35473.log.gz 2024-03-25T18:16:26.712 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41943.log 2024-03-25T18:16:26.712 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.40380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41631.log 2024-03-25T18:16:26.712 INFO:teuthology.orchestra.run.smithi066.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42060.log.gz 2024-03-25T18:16:26.712 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.41943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40497.log 2024-03-25T18:16:26.712 INFO:teuthology.orchestra.run.smithi066.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40380.log.gz 2024-03-25T18:16:26.712 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.41631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39990.log 2024-03-25T18:16:26.713 INFO:teuthology.orchestra.run.smithi066.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41943.log.gz 2024-03-25T18:16:26.713 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.40497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35446.log 2024-03-25T18:16:26.713 INFO:teuthology.orchestra.run.smithi066.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41631.log.gz 2024-03-25T18:16:26.713 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.39990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39170.log 2024-03-25T18:16:26.713 INFO:teuthology.orchestra.run.smithi066.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40497.log.gz 2024-03-25T18:16:26.713 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.35446.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39990.log.gz 2024-03-25T18:16:26.713 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T18:16:26.713 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.39170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39131.log 2024-03-25T18:16:26.713 INFO:teuthology.orchestra.run.smithi066.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.35446.log.gz 2024-03-25T18:16:26.713 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-osd.1.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39170.log.gz 2024-03-25T18:16:26.713 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41241.log 2024-03-25T18:16:26.713 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.39131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39092.log 2024-03-25T18:16:26.713 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.41241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40653.log 2024-03-25T18:16:26.713 INFO:teuthology.orchestra.run.smithi066.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39131.log.gz 2024-03-25T18:16:26.713 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.39092.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41241.log.gz 2024-03-25T18:16:26.713 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40968.log 2024-03-25T18:16:26.713 INFO:teuthology.orchestra.run.smithi066.stderr:gzip/var/log/ceph/ceph-client.admin.40653.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41280.log 2024-03-25T18:16:26.714 INFO:teuthology.orchestra.run.smithi066.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39092.log.gz 2024-03-25T18:16:26.714 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.40968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40302.log 2024-03-25T18:16:26.714 INFO:teuthology.orchestra.run.smithi066.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40653.log.gz 2024-03-25T18:16:26.714 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.41280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41085.log 2024-03-25T18:16:26.714 INFO:teuthology.orchestra.run.smithi066.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40968.log.gz 2024-03-25T18:16:26.714 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.40302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39444.log 2024-03-25T18:16:26.714 INFO:teuthology.orchestra.run.smithi066.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41280.log.gz 2024-03-25T18:16:26.714 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.41085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42138.log 2024-03-25T18:16:26.714 INFO:teuthology.orchestra.run.smithi066.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40302.log.gz 2024-03-25T18:16:26.714 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.39444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41709.log 2024-03-25T18:16:26.714 INFO:teuthology.orchestra.run.smithi066.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41085.log.gz 2024-03-25T18:16:26.714 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.42138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41124.log 2024-03-25T18:16:26.714 INFO:teuthology.orchestra.run.smithi066.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39444.log.gz 2024-03-25T18:16:26.714 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.41709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41163.log 2024-03-25T18:16:26.714 INFO:teuthology.orchestra.run.smithi066.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42138.log.gz 2024-03-25T18:16:26.714 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.41124.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41709.log.gz 2024-03-25T18:16:26.715 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40185.log 2024-03-25T18:16:26.715 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.41163.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T18:16:26.715 INFO:teuthology.orchestra.run.smithi066.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41124.log.gz 2024-03-25T18:16:26.715 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.40185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39912.log 2024-03-25T18:16:26.715 INFO:teuthology.orchestra.run.smithi066.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41163.log.gz 2024-03-25T18:16:26.715 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40419.log 2024-03-25T18:16:26.715 INFO:teuthology.orchestra.run.smithi066.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40185.log.gz 2024-03-25T18:16:26.715 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.39912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41670.log 2024-03-25T18:16:26.715 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.40419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39012.log 2024-03-25T18:16:26.715 INFO:teuthology.orchestra.run.smithi066.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39912.log.gz 2024-03-25T18:16:26.715 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.41670.log: gzip -5 86.2% --verbose -- replaced with /var/log/ceph/ceph-client.admin.40419.log.gz -- 2024-03-25T18:16:26.715 INFO:teuthology.orchestra.run.smithi066.stderr: /var/log/ceph/ceph-client.admin.39951.log 2024-03-25T18:16:26.715 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.39012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41514.log 2024-03-25T18:16:26.715 INFO:teuthology.orchestra.run.smithi066.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41670.log.gz 2024-03-25T18:16:26.715 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.39951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39405.log 2024-03-25T18:16:26.715 INFO:teuthology.orchestra.run.smithi066.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39012.log.gz 2024-03-25T18:16:26.716 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.41514.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39951.log.gz 2024-03-25T18:16:26.716 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41787.log 2024-03-25T18:16:26.716 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.39405.log: gzip 87.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.41514.log.gz -- 2024-03-25T18:16:26.716 INFO:teuthology.orchestra.run.smithi066.stderr: /var/log/ceph/ceph-client.admin.41592.log 2024-03-25T18:16:26.716 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.41787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39756.log 2024-03-25T18:16:26.716 INFO:teuthology.orchestra.run.smithi066.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39405.log.gz 2024-03-25T18:16:26.716 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.41592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40146.log 2024-03-25T18:16:26.716 INFO:teuthology.orchestra.run.smithi066.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41787.log.gz 2024-03-25T18:16:26.716 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40733.log 2024-03-25T18:16:26.716 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.39756.log: /var/log/ceph/ceph-client.admin.40146.log: 86.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41319.log 2024-03-25T18:16:26.716 INFO:teuthology.orchestra.run.smithi066.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41592.log.gz 2024-03-25T18:16:26.716 INFO:teuthology.orchestra.run.smithi066.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39756.log.gz 2024-03-25T18:16:26.716 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.40733.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29278.log 2024-03-25T18:16:26.716 INFO:teuthology.orchestra.run.smithi066.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40146.log.gz 2024-03-25T18:16:26.717 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41748.log 2024-03-25T18:16:26.717 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.41319.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40733.log.gz 2024-03-25T18:16:26.717 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph.tmp-client.admin.29278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41436.log 2024-03-25T18:16:26.717 INFO:teuthology.orchestra.run.smithi066.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29278.log.gz 2024-03-25T18:16:26.717 INFO:teuthology.orchestra.run.smithi066.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41319.log.gz 2024-03-25T18:16:26.717 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.41748.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T18:16:26.717 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41865.log 2024-03-25T18:16:26.717 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.41436.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41748.log.gz 2024-03-25T18:16:26.717 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35625.log 2024-03-25T18:16:26.717 INFO:teuthology.orchestra.run.smithi066.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41436.log.gz 2024-03-25T18:16:26.717 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.41865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42216.log 2024-03-25T18:16:26.717 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.35625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42177.log 2024-03-25T18:16:26.717 INFO:teuthology.orchestra.run.smithi066.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41865.log.gz 2024-03-25T18:16:26.717 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.42216.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35625.log.gz 2024-03-25T18:16:26.717 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41982.log 2024-03-25T18:16:26.717 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42216.log.gz 2024-03-25T18:16:26.718 INFO:teuthology.orchestra.run.smithi066.stderr: --/var/log/ceph/ceph-client.admin.42177.log: /var/log/ceph/ceph-client.admin.40772.log 2024-03-25T18:16:26.718 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.41982.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T18:16:26.718 INFO:teuthology.orchestra.run.smithi066.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42177.log.gz/var/log/ceph/ceph-client.admin.40772.log: 2024-03-25T18:16:26.718 INFO:teuthology.orchestra.run.smithi066.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41982.log.gz 2024-03-25T18:16:26.718 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39483.log 2024-03-25T18:16:26.718 INFO:teuthology.orchestra.run.smithi066.stderr: 86.4%/var/log/ceph/ceph-osd.0.log: -- replaced with /var/log/ceph/ceph-client.admin.40772.log.gz 2024-03-25T18:16:26.718 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41826.log 2024-03-25T18:16:26.718 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.39483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40458.log 2024-03-25T18:16:26.718 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.41826.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39483.log.gz 2024-03-25T18:16:26.718 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35144.log 2024-03-25T18:16:26.718 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40341.log 2024-03-25T18:16:26.718 INFO:teuthology.orchestra.run.smithi066.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41826.log.gz 2024-03-25T18:16:26.718 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.40458.log: /var/log/ceph/ceph-client.admin.35144.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:16:26.718 INFO:teuthology.orchestra.run.smithi066.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40458.log.gz 2024-03-25T18:16:26.718 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.40341.log: 90.0%gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T18:16:26.718 INFO:teuthology.orchestra.run.smithi066.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35144.log.gz 2024-03-25T18:16:26.719 INFO:teuthology.orchestra.run.smithi066.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40341.log.gz 2024-03-25T18:16:26.719 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39678.log 2024-03-25T18:16:26.719 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40575.log 2024-03-25T18:16:26.719 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.39678.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39678.log.gz 2024-03-25T18:16:26.719 INFO:teuthology.orchestra.run.smithi066.stderr: 92.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:16:26.719 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38763.log 2024-03-25T18:16:26.719 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42255.log 2024-03-25T18:16:26.719 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.40575.log: /var/log/ceph/ceph-client.admin.38763.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40575.log.gz 2024-03-25T18:16:26.719 INFO:teuthology.orchestra.run.smithi066.stderr: 87.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39639.log 2024-03-25T18:16:26.719 INFO:teuthology.orchestra.run.smithi066.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38763.log.gz 2024-03-25T18:16:26.719 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.42255.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42255.log.gz 2024-03-25T18:16:26.719 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39327.log 2024-03-25T18:16:26.719 INFO:teuthology.orchestra.run.smithi066.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T18:16:26.719 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.39639.log: /var/log/ceph/ceph-client.admin.40614.log 2024-03-25T18:16:26.719 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.39327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39053.log 2024-03-25T18:16:26.720 INFO:teuthology.orchestra.run.smithi066.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39639.log.gz 2024-03-25T18:16:26.720 INFO:teuthology.orchestra.run.smithi066.stderr: 86.6%/var/log/ceph/ceph-client.admin.40614.log: -- replaced with /var/log/ceph/ceph-client.admin.39327.log.gz 2024-03-25T18:16:26.720 INFO:teuthology.orchestra.run.smithi066.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39795.log 2024-03-25T18:16:26.720 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.39053.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T18:16:26.720 INFO:teuthology.orchestra.run.smithi066.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40614.log.gz 2024-03-25T18:16:26.720 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-client.admin.39795.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39053.log.gz 2024-03-25T18:16:26.720 INFO:teuthology.orchestra.run.smithi066.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39795.log.gz 2024-03-25T18:16:26.768 INFO:teuthology.orchestra.run.smithi193.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T18:16:27.508 INFO:teuthology.orchestra.run.smithi066.stderr:/var/log/ceph/ceph-mon.c.log: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T18:16:27.512 INFO:teuthology.orchestra.run.smithi193.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T18:16:27.528 INFO:teuthology.orchestra.run.smithi066.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T18:16:27.556 INFO:teuthology.orchestra.run.smithi193.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T18:16:27.609 INFO:teuthology.orchestra.run.smithi193.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T18:16:27.789 INFO:teuthology.orchestra.run.smithi066.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T18:16:27.796 INFO:teuthology.orchestra.run.smithi066.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T18:16:27.835 INFO:teuthology.orchestra.run.smithi193.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T18:16:31.760 INFO:teuthology.orchestra.run.smithi066.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T18:16:31.929 INFO:teuthology.orchestra.run.smithi193.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T18:16:31.931 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-25T18:16:31.931 INFO:teuthology.orchestra.run.smithi193.stderr:real 0m5.259s 2024-03-25T18:16:31.931 INFO:teuthology.orchestra.run.smithi193.stderr:user 0m8.788s 2024-03-25T18:16:31.931 INFO:teuthology.orchestra.run.smithi193.stderr:sys 0m0.318s 2024-03-25T18:16:32.036 INFO:teuthology.orchestra.run.smithi066.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T18:16:32.038 INFO:teuthology.orchestra.run.smithi066.stderr: 2024-03-25T18:16:32.038 INFO:teuthology.orchestra.run.smithi066.stderr:real 0m5.393s 2024-03-25T18:16:32.038 INFO:teuthology.orchestra.run.smithi066.stderr:user 0m13.929s 2024-03-25T18:16:32.039 INFO:teuthology.orchestra.run.smithi066.stderr:sys 0m0.517s 2024-03-25T18:16:32.039 INFO:tasks.ceph:Archiving logs... 2024-03-25T18:16:32.039 DEBUG:teuthology.misc:Transferring archived files from smithi066:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621249/remote/smithi066/log 2024-03-25T18:16:32.040 DEBUG:teuthology.orchestra.run.smithi066:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:16:34.328 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621249/remote/smithi193/log 2024-03-25T18:16:34.329 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:16:35.805 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:16:36.006 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=1bf35a3674af4a11b1df78057fd6fede Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:16:36.011 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T18:16:36.021 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T18:16:36.035 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T18:16:36.035 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:16:36.048 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:16:36.128 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T18:16:36.138 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T18:16:36.138 DEBUG:teuthology.orchestra.run.smithi066:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:16:36.140 DEBUG:teuthology.orchestra.run.smithi193:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:16:36.154 INFO:teuthology.orchestra.run.smithi066.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:16:36.157 INFO:teuthology.orchestra.run.smithi193.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:16:36.160 INFO:teuthology.orchestra.run.smithi066.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:16:36.160 INFO:teuthology.orchestra.run.smithi066.stdout:=============================================================================== 2024-03-25T18:16:36.161 INFO:teuthology.orchestra.run.smithi066.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 44 +616us[ +616us] +/- 91ms 2024-03-25T18:16:36.161 INFO:teuthology.orchestra.run.smithi066.stdout:^* hv02.front.sepia.ceph.com 3 6 377 47 -1237us[-1243us] +/- 73ms 2024-03-25T18:16:36.161 INFO:teuthology.orchestra.run.smithi066.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 +829us[ +829us] +/- 78ms 2024-03-25T18:16:36.161 INFO:teuthology.orchestra.run.smithi066.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:16:36.162 INFO:teuthology.orchestra.run.smithi193.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:16:36.162 INFO:teuthology.orchestra.run.smithi193.stdout:=============================================================================== 2024-03-25T18:16:36.163 INFO:teuthology.orchestra.run.smithi193.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 49 +617us[ +628us] +/- 91ms 2024-03-25T18:16:36.163 INFO:teuthology.orchestra.run.smithi193.stdout:^* hv02.front.sepia.ceph.com 3 6 377 47 -1191us[-1180us] +/- 73ms 2024-03-25T18:16:36.163 INFO:teuthology.orchestra.run.smithi193.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 48 +830us[ +842us] +/- 78ms 2024-03-25T18:16:36.163 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:16:36.164 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T18:16:36.173 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T18:16:36.173 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T18:16:36.183 DEBUG:teuthology.orchestra.run.smithi066:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:16:36.280 DEBUG:teuthology.orchestra.run.smithi193:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:16:36.352 DEBUG:teuthology.orchestra.run.smithi066:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:16:36.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:16:36.388 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:16:36.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:16:36.419 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T18:16:36.428 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T18:16:36.437 INFO:teuthology.task.internal:Duration was 7299.814233 seconds 2024-03-25T18:16:36.438 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T18:16:36.449 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T18:16:36.449 DEBUG:teuthology.orchestra.run.smithi066:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:16:36.451 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:16:36.501 INFO:teuthology.orchestra.run.smithi066.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:16:36.506 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:16:36.859 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T18:16:36.860 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi066.front.sepia.ceph.com 2024-03-25T18:16:36.860 DEBUG:teuthology.orchestra.run.smithi066:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:16:36.881 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi193.front.sepia.ceph.com 2024-03-25T18:16:36.881 DEBUG:teuthology.orchestra.run.smithi193:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:16:36.902 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T18:16:36.903 DEBUG:teuthology.orchestra.run.smithi066:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:16:36.924 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:16:36.974 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T18:16:36.974 DEBUG:teuthology.orchestra.run.smithi066:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:16:36.997 DEBUG:teuthology.orchestra.run.smithi193:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:16:37.119 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T18:16:37.129 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T18:16:37.129 DEBUG:teuthology.orchestra.run.smithi066:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:16:37.159 DEBUG:teuthology.orchestra.run.smithi193:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:16:37.187 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T18:16:37.196 DEBUG:teuthology.orchestra.run.smithi066:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:16:37.202 DEBUG:teuthology.orchestra.run.smithi193:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:16:37.228 INFO:teuthology.orchestra.run.smithi066.stdout:kernel.core_pattern = core 2024-03-25T18:16:37.252 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = core 2024-03-25T18:16:37.282 DEBUG:teuthology.orchestra.run.smithi066:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:16:37.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:16:37.317 DEBUG:teuthology.orchestra.run.smithi193:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:16:37.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:16:37.336 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T18:16:37.345 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T18:16:37.345 DEBUG:teuthology.misc:Transferring archived files from smithi066:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621249/remote/smithi066 2024-03-25T18:16:37.345 DEBUG:teuthology.orchestra.run.smithi066:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:16:37.397 DEBUG:teuthology.misc:Transferring archived files from smithi193:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621249/remote/smithi193 2024-03-25T18:16:37.397 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:16:37.432 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T18:16:37.432 DEBUG:teuthology.orchestra.run.smithi066:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:16:37.434 DEBUG:teuthology.orchestra.run.smithi193:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:16:37.482 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T18:16:37.493 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T18:16:37.493 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T18:16:37.504 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T18:16:37.504 DEBUG:teuthology.orchestra.run.smithi066:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:16:37.507 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:16:37.519 INFO:teuthology.orchestra.run.smithi066.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:16 /home/ubuntu/cephtest 2024-03-25T18:16:37.536 INFO:teuthology.orchestra.run.smithi193.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:16 /home/ubuntu/cephtest 2024-03-25T18:16:37.537 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T18:16:37.546 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T18:16:37.568 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T18:16:37.593 DEBUG:teuthology.nuke:shortname: smithi066 2024-03-25T18:16:37.594 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:16:37.616 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi066.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621249', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:02:22.837681', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHiFRiyARlFOX6EzyMAmT7g0HngrubTPU4jsSfsyWrlEflazx4IQRo9ara2llBy4NEOGryTG/sLr+mPufh99/n8='} 2024-03-25T18:16:37.619 DEBUG:teuthology.nuke:shortname: smithi193 2024-03-25T18:16:37.619 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:16:37.642 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621249', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:02:22.835849', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDCcALhUKDoQ73A9zVvrTXTBClHg7qAmouPpjds8k2OzS4rF5FKG4D5lDjy5mM8V+0RU7Lck9wnVivw9YWcRFI8='} 2024-03-25T18:16:37.666 INFO:teuthology.orchestra.console.smithi066:Power off 2024-03-25T18:16:37.666 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:16:37.689 DEBUG:teuthology.orchestra.console.smithi066:power off output: Chassis Power Control: Down/Off 2024-03-25T18:16:37.689 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:16:37.707 INFO:teuthology.orchestra.console.smithi193:Power off 2024-03-25T18:16:37.707 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:16:37.716 DEBUG:teuthology.orchestra.console.smithi066:check power output: Chassis Power is on 2024-03-25T18:16:37.729 DEBUG:teuthology.orchestra.console.smithi193:power off output: Chassis Power Control: Down/Off 2024-03-25T18:16:37.729 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:16:37.751 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2024-03-25T18:16:41.717 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:16:41.752 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:16:41.828 DEBUG:teuthology.orchestra.console.smithi066:check power output: Chassis Power is on 2024-03-25T18:16:41.861 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2024-03-25T18:16:45.834 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:16:45.862 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:16:45.944 DEBUG:teuthology.orchestra.console.smithi066:check power output: Chassis Power is on 2024-03-25T18:16:45.971 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2024-03-25T18:16:49.949 DEBUG:teuthology.orchestra.console.smithi066:pexpect command: ipmitool -H smithi066.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:16:49.972 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:16:50.058 DEBUG:teuthology.orchestra.console.smithi066:check power output: Chassis Power is off 2024-03-25T18:16:50.080 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is off 2024-03-25T18:16:50.160 INFO:teuthology.orchestra.console.smithi066:Power off completed 2024-03-25T18:16:50.181 INFO:teuthology.orchestra.console.smithi193:Power off completed 2024-03-25T18:16:50.286 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 7299.81423330307 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=1bf35a3674af4a11b1df78057fd6fede status: fail success: false 2024-03-25T18:16:50.286 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:16:50.367 INFO:teuthology.run:FAIL